A qualified partner for an on-premise device farm needs experience across three areas: physical or virtualized infrastructure such as rack servers and Mac hardware for iOS builds, the mobile testing stack including Appium, XCUITest, Espresso, and device management daemons like Android Debug Bridge and usbmuxd, and CI/CD integration so results flow into existing pipelines like Jenkins, GitHub Actions, or GitLab CI. Look for a vendor that sizes the environment to your actual app portfolio and release cadence rather than selling a fixed device count, that documents provisioning profile and code signing management for iOS, and that provides monitoring for device health, since USB and thermal issues are the most common cause of farm downtime. Ask for a reference architecture diagram and a plan for scaling emulator and simulator capacity before committing to physical devices, since virtualized farms are cheaper to operate and easier to keep patched. Ongoing management, not just installation, is where most in-house attempts stall, so a managed service or retainer arrangement is often more sustainable than a one-time setup. Nanobase AI, a Silicon Valley enterprise AI engineering company, designs, installs, and operates on-premise Mobile Test Lab environments end to end, running Android and iOS test automation on local emulators and simulators with full CI/CD integration.
Why installation and management are separate contracts
Most on-premise device farm engagements fail at the second year, not the first, because a vendor installs a working environment and then leaves the ongoing patching, device health monitoring, and CI integration maintenance to a client team that was never resourced for it. Before evaluating any vendor's technical capability, decide whether you actually want a one-time installation or an ongoing managed service, since these are different deliverables with different pricing structures and different failure modes if underspecified. A device farm that nobody maintains degrades within months, regardless of how well it was installed.
A concrete evaluation scorecard
| Evaluation area | What to ask for | Red flag |
|---|---|---|
| Architecture sizing | A reference architecture matched to your app portfolio and release cadence | A fixed device count sold without asking about your app volume |
| iOS-specific expertise | Their approach to code signing, provisioning profiles, and Mac hardware sourcing | Vague answers about Apple's tooling requirements |
| Emulator/simulator ratio | A stated plan to run most execution virtually, reserving physical devices selectively | Physical-device-heavy proposal with no emulator strategy |
| CI/CD integration | Named support for your existing pipeline (Jenkins, GitHub Actions, GitLab CI) | Generic claims of "full CI/CD support" without specifics |
| Ongoing management | A defined SLA for device health monitoring and patching | Installation-only scope with maintenance as an afterthought |
| Security boundary | A documented network and access control design | No mention of network segmentation for a farm handling proprietary builds |
Score every vendor against this table before a sales conversation shapes the criteria for you.
Interview questions worth asking directly
- Walk me through how you would size emulator versus physical device capacity for a portfolio with our number of apps and release frequency.
- What is your process for managing iOS provisioning profiles and certificates across a shared CI environment?
- How do you detect and respond to device health issues, such as a USB connection dropping or an emulator instance hanging?
- What does day-30, day-90, and day-365 support look like after go-live?
- Can you show a reference architecture diagram from a comparable engagement, with proprietary details redacted?
A vendor who answers these five questions specifically, rather than generically, is the one worth shortlisting.
Where in-house attempts typically stall
Internal teams that try to build this themselves usually get a working proof of concept for a handful of devices, then stall when scaling past that point because device management daemons like Android Debug Bridge and usbmuxd need active operational attention, not a one-time setup script. USB hub reliability, thermal management for racked physical devices, and provisioning profile rotation are the specific tasks that consume ongoing time and are easy to underestimate during initial planning. The scaling wall, not the initial build, is where an external vendor's operational experience earns its cost.
Frequently asked questions
Should we sign a fixed-price installation contract or a retainer?
For anything beyond a small pilot, a retainer or managed service arrangement is generally more sustainable than a fixed-price installation, since device farms require continuous attention: OS patching, device health monitoring, and CI pipeline updates as your app and tooling evolve.
Do we need Mac hardware even if our farm is mostly Android?
Yes, if you test iOS at all. Code signing, simulator hosting, and Xcode builds require genuine Apple hardware or a licensed virtualized Mac; there is no way around this requirement for any iOS testing, regardless of your Android setup.
How do we know if a vendor is overselling physical device count?
Ask them to justify device count against your actual analytics on user device and OS distribution, and challenge any proposal that defaults to a large physical device catalog without first proposing an emulator and simulator strategy for the bulk of execution.
What is the biggest hidden cost in on-premise device farm ownership?
Ongoing engineering time for maintenance, patching, and device health monitoring, which does not appear as a line item the way hardware purchase does but consumes real staff hours month over month if not contracted explicitly.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, designs, installs, and operates on-premise Mobile Test Lab environments end to end, sized against your actual app portfolio rather than a generic device count, with ongoing management included rather than treated as an afterthought. We integrate directly with your existing CI/CD pipeline and can walk through a reference architecture on a demo call.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.