S3 and Django Staticfiles collectstatic command upload only changed files

I just revisited a problem getting S3 and collectstatic to play nicely via django-storages.

I spent an hour wondering what had changed in django or django-storages that started to force the collectstatic command to always upload all images.

I did a line by line code comparison between my two django and storages installations and couldn’t find anything odd.

I started to google more.

Hello, me!

I ended up on this site: http://c4urself.posterous.com/djangos-collectstatic-with-s3boto which actually linked to a few of my original posts.

For the record this happens to me on a daily basis (my memory sucks) but this was rather unexpected in the vastness of the world wide internets.

You MUST install python-dateutill==1.5

Collectstatic will silently fail trying to detect the modified time of the S3 files, and consequently will always upload a new file. Since there was no error, I had no idea something was failing until I read the above post which pointed to the specific function (mentioned by me originally, apparently).

If your collectstatic command is always uploading all files, make sure you have python-dateutil==1.5 installed!

‘str’ object has no attribute ‘get’

Got this error when accidentally returning a string instead of an HttpResponse class in an admin view.

Shopify JSON API example using Python Requests

Shopify API XML and JSON example using Python Requests

I didn’t find any full examples of using the Shopify API in either XML or JSON.

I tried using the Shopify Python library but had trouble identifying the currently saved one to many objects (the Product Variants).

I could easily upload NEW variants, but I could not tell which python object variants received which shopify IDs which I absolutely needed as I was using the API for two way synchronization (pushing and pulling changes).

To put the long story short, the culprit was not having the correct Content-Type header for PUT and POST requests.

Using text/json, shopify was returning an unhelpful 500 error with the error message: “Errors: error” – not helpful! I started wondering if I was using the wrong urls… their template suggesting admin/#{id}.json was a bit confusing too. Why not just write admin/{id}.json ?

Set up authentication

Using the python requests library makes this extremely easy.

request = requests.Session(auth=(settings.SHOPIFY_API_KEY, settings.SHOPIFY_API_PASSWORD))
print json.loads(request.get('http://myshop.myshopify.com/admin/assets.json').content)

Create a product

payload = '''{
	  "product": {
	    "body_html": "<strong>Good snowboard!</strong>",
	    "product_type": "Snowboard",
	    "title": "Burton Custom Freestlye 151",
	    "variants": [
	      {
	        "price": "10.00",
	        "option1": "First"
	      },
	      {
	        "price": "20.00",
	        "option1": "Second"
	      }
	    ],
	    "vendor": "Burton"
	  }
	}'''

response = request.post('http://myshop.myshopify.com/admin/products', 
	data=payload,
	headers={'
		'Content-Type': 'application/json', # this is the important part.
	},)
print response.status_code, response.content

Modify an existing product

payload = '''{
	  "product": {
	    "published": false,
	    "id": 632910392
	  }
	}'''
response = request.put('http://myshop.myshopify.com', data=payload, headers={'Content-Type': 'application/json'})

Shopify Behind an Nginx Reverse Proxy

For SEO Purposes and a general move away from the Shopify platform, we at Grove have finally implemented a reverse proxy via Nginx.

Previously, our DNS records for www.grovemade.com pointed directly at grove.myshopify.com, while team.grovemade.com pointed to our linode.com VPS. The problem with this approach is SEO – search engines rank subdomains as separate entities. team.grovemade.com is competing with www.grovemade.com.

Let’s face it – at the end of the day, Shopify is an amazingly useful platform. It does as well as a general solution can. We used the best of both worlds: Shopify would serve the e-commerce pages, and Linode would serve our custom django project.

The solution to our problem? Enter the proxy server.

Set the DNS records to point all traffic to grovemade.com to our nginx server at linode, and have the nginx server proxy specific URLs to Shopify and the rest to our linode servers.

That means when you access www.grovemade.com/collections/foobar, nginx proxies the request to grove.myshopify.com and returns the data to your browser seamlessly.

When you access www.grovemade.com/foobar/, nginx proxies the request to a local apache server hosting our django project.

Nginx proxy configuration

Here’s the configuration. It was pretty painless once I realized I could proxy to a subdomain already mapped to Shopify.

Note that if you do not proxy_pass to a domain Shopify knows about via the Shopify admin DNS settings you must manually set the Host parameter via nginx `proxy_set_header Host mystore.myshopify.com`

# index url
# ---------
location = / {
    proxy_pass http://shopify.grovemade.com;

    client_max_body_size    10m;
    client_body_buffer_size     128k;
    proxy_connect_timeout 90;
}


# grove urls
# ----------
location / {
    proxy_pass http://127.0.0.1:8080/;
    proxy_redirect off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90; # time to connect to upstream server
    proxy_send_timeout         120; # time to wait for upstream to accept data
    proxy_read_timeout         120; # time to wait for upstream to return data

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;
}




# shopify urls
# ------------
location ~ ^/(collections|cart|products|shopify|pages|blogs|checkout|admin)/? {
    proxy_pass http://shopify.grovemade.com;

    client_max_body_size    10m;
    client_body_buffer_size     128k;
    proxy_connect_timeout 90;
}

Restart your nginx server and watch your traffic proxied!

Extra useful stuff you can do when you share the same domain

When your django servers and Shopify share the same domain name, you get more than just SEO. You get access to the Shopify cookies… which means we can programmatically make requests to Shopify to enter checkout, or read the contents of the cart.

I’ve just started to mess around with this, but it appears that Shopify sets a `cart` cookie with an ID string that you can easily read in django via `request.COOKIES.get(‘cart’)` .

Add this to the headers when you make a GET request or POST request, and you can manually enter checkout, use the “js” API from python, etc. We’ll be using this to literally only use their checkout page for our site.

Django — dictionary update sequence element #0 has length 1; 2 is required

This error appears if you accidentally pass a string to the url function.

I was trying to name my url and forgot it’s a kwarg.

url(r'^$', 'foo', 'this_is_invalid'),

Bash — Escape spaces in results of `pwd` command

Just wrap the `pwd` in quotes to escape spaces. It’s just literally wrapping the output of pwd in a string “”.

$ > scp `pwd`/my_file.png foo # reads pwd spaces as argument separator
$ > scp "`pwd`"/my_file.png foo 

When Python Packages Time Out (PIP / easy_install) use mirrors

http://jacobian.org/writing/when-pypi-goes-down/

pip install –use-mirrors $PACKAGE

Django Storages, Boto, and Amazon S3 Slowness on manage.py collectstatic command fixed

I’ve had a few issues moving everything to Amazon S3.

First, django’s collectstatic management command wasn’t even detecting modified files, so the command would re-upload all files to amazon every invocation.

Django-Storages v1.1.3 fixed this problem, but now I noticed a new problem: modified files were taking less time to detect, but still far too long given that one call was returning the meta data Amazon S3.

After some digging, I found the problem in the modified_time method where the fallback value is being called even if it’s not being used. I moved the fallback to an if block to be executed only if get returns None

entry = self.entries.get(name, self.bucket.get_key(self._encode_name(name))) 
# notice the function being called to populate the default value, regardless
# of whether or not a default is required.

That code should be wrapped in an if statement and only fire the expensive function if the get function fails.

    entry = self.entries.get(name)
    if entry is None:
        entry = self.bucket.get_key(self._encode_name(name))

This change spurred me to create a bitbucket account and learn some basic Mercurial commands : )

Hope it helps somebody else out there with many more than 1k files to sync.

The results

I benchmarked the difference on my local machine between the two functions.

For 1000 files, the new version took less than .1s vs the old function which took 11.5 seconds.

https://bitbucket.org/yuchant/django-storages/overview

Python — add suffix if string slice truncated contents

A common design pattern: truncate a string to X chars if the string is long, and show a ‘…’ to indicate the string has been truncated.

What’s the shortest way to write this code?

http://stackoverflow.com/questions/2872512/python-truncate-a-long-string

x[:10] + (x[:10] and '...')

jQuery UI Draggable Table Rows Gotchas

I ran into a few Gotchas while dragging table rows. One was an easy fix, the other took some guesswork.

Can’t drag individual TR elements

If you set a “TR“ element to “draggable()“, it will not work because we don’t know how to handle a moving TR element (TRs must be in a table to render properly).

To fix this, wrap the TR in a table temporarily while dragging via the helper option.

            $(".tr_row").draggable({
                helper: function(event) {
                    return $('<div class="drag-row"><table></table></div>')
  .find('table').append($(event.target).closest('tr').clone()).end();
                },
            });

Sortable / Droppable doesn’t work on TR elements, or TR elements disappear during drag

The second problem I ran into was a little more obscure. Your droppable and sortable selectors MUST be applied to a “<tbody>“ tag.

I noticed that when I had the normal “<table><tr><td>“ structure and dragged, jQuery created a <tbody> element to contain the rows.

 $("#my_selector tbody").droppable().draggable();