Mass Migrations: Moving Customers, Orders and Products Over to Shopify and Shopify Plus
Migrating data, customers, order history and products—over the years we’ve never found a solid overview that properly documents and explains the challenges, pitfalls, and best practices, particularly when it comes to huge amounts of data. So, we thought we’d do our best to cover it all in this article.
Shopify is in and it’s in for a reason.
Knowing that all of your current and future transactions will be secure, hosting is managed professionally, themes are both beautiful and functional, and that you can have a support staff at hand around the clock – these are all enough to make most e-commerce people ditch their old platform outright.
So here you are, ready to switch your existing store over to your new Shopify installation and suddenly, a little voice in the back of your head starts squealing, “Wouldn’t it make sense to migrate all of your order and user data so that you have some accountability on the new platform without having to check out your backups?”
Wow. That was a pretty solid idea, voice-in-the-back-of-your-head. Good thinking!
To be fair, it often makes sense to start over from scratch. Clean starts are wonderful in that they allow you to adjust to the new structure on your own time, knowing that there is good reason to keep and hold on to a backup of your old data. And, you really should be keeping backups of your old site regardless. But you can’t always start fresh, be it through client request or your own internal needs.
There are services that will do it for you, of course. But if your old site was totally custom built or if you were on a less commonly used e-commerce platform, you could be stuck with having to do it yourself. That, or if you just want accountability and the means to iterate over data formatting so that you can be 100% sure everything is workable with whatever internal processes you may have. Or heck, maybe you just need to mass-import metafield data. No judgment here!
The good news is that Shopify has an API that we can use to accomplish these goals. Below is some basic guidance, suggestions, and gotcha’s that we’ve accumulated while working through our own implementation of some mass-import tools.
Ways to Upload Things
Because Shopify’s API is RESTful and HTTP-request based, you can use whatever language you want to interact with it so long as it can handle sending and receiving HTTP requests. We started out using Javascript for the uploads and Python to process data, but eventually switched to building out full Node.js implementations that have the ability to hook up to an existing database, format the data, and upload it all by itself.
Before starting with your own implementation, you should take a look at Shopify’s take on using the Postman Tool. Not only will this tool allow you to interact with the API using a GUI interface, but it will also let you export basic code for you to use within your future implementation. If you’re a developer with limited RESTful API experience, then this step is a must. Considering that you’ll have to obtain API credentials outright for any non-browser-based solution, this makes testing things with Postman a solid first step regardless of your experience.
From here you should determine what and how you want to get data imported. Are you doing a simple translation from one dataset to another? How much data have you got to upload? Are you looking to do this multiple times or just once? Believe it or not, the credentials you get from simply logging in to the admin portion of your Shopify are enough to make successful API requests from directly within your browser. However, if you’re interacting with large sets of data, we’d wholly recommend that you build your own custom app to take care of these things.
If you want to go the in-browser route and you’re only planning on importing a couple hundred redirects or orders, then you can write javascript directly into your console to accomplish the task. Why spend extra time building out a tool when you know that the scope is already pretty small? Plus, you can always take this code one step further by reusing it to build out a bookmarklet. As alluded to above, a perfect situation for this approach would be to add 301 redirects – you’ll only be uploading these once, and the data required is relatively lightweight.
Our team did its first iteration on internal tools for mass importing content with a javascript bookmarklet. This took advantage of the above use-case, essentially creating a GUI for running simple javascript in-browser. At the time this made sense – we only had to import a small amount of orders and redirects and our sources were so varied, changing, and inconsistent that it was easier to hand-massage that data into usable content with python scripts first before uploading anything.
That changes pretty quickly once you have to import a substantial amount of data or start working with a client who has very specific data migration needs. Browsers have memory limitations after all, so queueing up tens of thousands of API calls will start to cause some issues. After all, you don’t want to babysit your uploads, do you?!
Our first iteration of the Node.js App solves those problems and this should be the de-facto direction for anyone already comfortable with writing javascript. Our app uses the Request and Request Promise libraries to make HTTP calls and the MySQL library to deal with querying and importing data for manipulation. It currently runs as a simple script, but because of the chosen environment, it’s easy enough to flesh out the system with a GUI when there is time to invest.
Published Limitations
Shopify has done a pretty solid job of documenting the basics of their Admin API. You can find all of that documentation here.
One thing that you should know up-front is that when you’re bulk importing items, the Shopify API requires you to do each item individually. This puts the onus of dealing with rate limiting, error collection, and organizing data into your hands.
Shopify, out of the box, will allow for around two calls every second. They use a “leaky bucket” algorithm to control and limit upload rates with a bucket size of 40 calls. The bucket has a “leak rate” of two calls per second, so technically you could make 40 calls near-simultaneously without issue, but would then have to limit all calls going forwards as the “leaky bucket” is full and not emptying quick enough to continue at the initial rate. There are dozens of rate limiting libraries that you can use that will allow gradually increasing the time between calls, so knowing the scope of your project will influence your approach to this aspect. If you’re working with a Shopify Plus site, your site manager may extend an offer to increase your bucket size and leak rate.
When working on our browser-based solutions, we started with using XMLHttpRequests for hitting the API’s endpoints. Later needing to authenticate for uploading Shopify Plus items forced us to switch to using Fetch to manage our asynchronous uploads. That allowed us to send the correct headers without any extraneous browser-related data. The idea was to originally build something out as bare-boned as possible in an effort to make this tool compatible across a range of browsers. However, client needs often trump personal development so we put that on the back-burner.
As it stands, most of the Shopify Admin APIs limitations are pretty manageable and easy enough to work around.
Unpublished Limitations
There are a few more things worth knowing before jumping into building your own bulk uploader for Shopify. We came across a few of these “gotchas” after working through our own implementations. Of course, just like everything else in tech, Shopify’s API is somewhat flux and some of these issues may change or have been resolved through an updated API by the time you get around to reading this entry so never hesitate to consult the documentation.
When you get credentials for your app through the Shopify Private App system, you’ll eventually find yourself in front of a checkbox for enabling the “Storefront API.” This API enables features far outside of the scope of a simple bulk uploader, but if you do happen to find yourself with this enabled, you’ll need to know that regardless of how you set up your importable order data, confirmation emails will be sent out to the order’s associated email. This is a big issue if you’re importing legacy data and want to avoid triggering panic amongst your clientele. Of course, there are only a select few scenarios where you’d be mass-importing data and using the Storefront API in the same app, so it would make sense to leave this box unchecked until you absolutely need to enable the API.
Also, before you upload, your data you should take care to double check emails and phone numbers. We’ve experienced symptoms that suggest Shopify runs advanced checks on both of those fields, so it’s best to check for email and phone validity with plugins that are capable of doing more than just simple syntax checks. In one of our efforts to massage data into a directly-in-the-admin-importable CSV for customers, we’ve used the “phonenumbers” and “validate_email” python plugins. This has the dual benefits of both limiting the number of errors generated on upload and cleaning out some broken/extraneous data.
Also, make sure you understand JSON and how the objects you’re uploading are expected to be formatted. You’ll see examples in the documentation where some information is included on the demo object while some information isn’t – do what you can to include everything. The relationships are published, but taking flexibility for granted can land you with some serious headaches. For example, the ‘accepts_marketing’ key on an Order object is set as ‘false’ by default, overriding whatever is already set on the existing user account. This means that if you upload accounts separately from orders, you very well may override some customer data by excluding that information from your uploaded JSON. The default value is indicated as a note in the documentation, but many of the examples exclude that field so be sure to double check the documentation instead of gleaning meaning from the examples.
And finally, speaking on errors: you can usually expect to see some relatively descriptive error messages in the response. However, if you misformat data that you upload you won’t get the same luxury. This means that if you find yourself banging your head against the keyboard because you’ve suddenly lost descriptive error responses, double check how the more complex portions of your JSON objects are formatted. If the API expects an array with other objects inside it, don’t send it an array with strings in it. Pretty straight-forward, right? Unfortunately, sometimes it takes making stupid mistakes to confirm just how straightforward things truly are.
Conclusion
All in all, building a mass-import tool for your Shopify site only looks daunting on paper. More often than not, interpreting your client’s data will be the hardest part of the import. Thankfully, the perceived burden is easily resolved by the flexibility of javascript. Need to look up content from one table in the other, but don’t want to join? Just build a hash table! Want to do a quick import of 301 redirects? Do it in your browser’s console! Need to upload 100k items with specific and predictable formatting? Build out a node app. Hopefully, after a little bit of fiddling, you too will find world provided by Shopify’s Admin API as welcoming as any other.
********
A Shopify Plus Solutions Partner, Sleepless Media has been designing and developing e-commerce websites for over 15 years. We’ve been explicitly focused on Shopify’s powerful platform since 2010 and have migrated countless stores from other platforms such as Magento, Volusion, 3dCart, osCommerce, Woocommerce, Bigcommerce and more. Have something you want to run by us? Get in touch.
