Django and Scrapy

I'm currently working on a project which centres around pulling in data from an external website, "mashing" it up with some additional content, and then displaying it on a website.

The website is going to be interactive and reasonably complex so I decided to use django. To acquire the external data there isn't a webservice so I'm stuck parsing html (and excel spreadsheets but that's a separate story). Scrapy seemed ideal for this and although I wish I had used some other approach than xpath it largely has been.

Having set up my database models in django and built my spider in scrapy the next step was putting the data from the spider in the database. There are plenty of posts detailing how to use the django ORM from outside a django project, even some specific to scrapy but they didn't seem to be working for me.

The issue was the way I handled development and production environment settings.

Continue reading ...

Numpy talk at Python Northwest

Back in December I gave a talk introducing Numpy to the PyNorthwest group. The slides are available as a pdf.

Although I frequently use Numpy I'm far from an expert and the content of my talk reflected this. I started with a general introduction to the array object and then expanded the scope of the talk to highlight some of the projects that use Numpy. I gave an example of using MDP and matplotlib.

The talk was followed by some excellent discussion. We went through some of the code on slide 6 in a lot of detail.

The PyNorthwest group meets at Madlab in Manchester city centre on the third Thursday of each month. If you're in the area check it out. The January event is on the 19th, starting at 7pm.

Full text visualisation

At BarcampNortheast4 last weekend and at the Python Northwest meetup on Thursday I gave a presentation on the work I've been doing generating full text visualisations of PDF document libraries.

This was the third BarcampNortheast event I have attended. Each has been slightly different but they have all been a weekend well spent. This year felt a little smaller than previous years but that may have partly been because we were in a bigger space.

I have been attending the python Edinburgh meetups for a while. They have always been interesting and the Northwest meetup this Thursday was the first since I moved back to the Northwest. The format, alternating talks and coding sessions, is different to Edinburgh, regular pub meetups with irregular talks, coding sessions and miniconferences. It was an interesting crowd and the other talks, on Apache Thrift and teaching programming to GCSE students (15-16 year olds), gave a really good variety of subjects to discuss later.

Continue reading ...

Images and Vision in Python: Slides from talk at Python Edinburgh Mini-Conf 2011

Last weekend the Python Edinburgh users group hosted a mini-conference. Saturday morning was kicked off with a series of talks followed by sessions introducing and then focusing on contributing to django prior to sprints which really got going on the Sunday.

The slides for my talk on, "Images and Vision in Python" are now available in pdf format here.

The slide deck I used is relatively lightweight with my focus being on demonstrating using the different packages available. The code I went through is below.

from PIL import Image

#Open an image and show it
pil1 ='filename')

#Get its size
pil1s = pil1.resize((100,100))
#or - thumbnail
pil1.thumbnail((100,100), Image.ANTIALIAS)

#New image
bg ='RGB', (500,500), '#ffffff')

#Two ways of accessing the pixels
#getpixel/putpixel and load
#load is faster
pix = bg.load()

for a in range(100, 200):
	for b in range(100,110):
		pix[a,b] = (0,0,255)

#Drawing shapes is slightly more involved
from PIL import ImageDraw
draw = ImageDraw.Draw(bg)
draw.ellipse((300,300,320,320), fill='#ff0000')

from PIL import ImageFont
font = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeSerif.ttf", 72)
draw.text((10,10), "Hello", font=font, fill='#00ff00')

#Demo's for vision
from scipy import ndimage
import mahotas

#Create a sample image
v1 = np.zeros((10,10), bool)
v1[1:4,1:4] = True
v1[4:7,2:6] = True
imshow(v1, interpolation="Nearest")
imshow(mahotas.dilate(v1), interpolation="Nearest")
imshow(mahotas.erode(v1), interpolation="Nearest")
imshow(mahotas.thin(v1), interpolation="Nearest")

#Opening, closing and top-hat as combinations of dilate and erode

#Latest version of mahotas has a label func
v1[8:,8:] = True
labeled, nr_obj = ndimage.label(v1)
imshow(labeled, interpolation="Nearest")

#Convert a grayscale image to a binary image
v2 = mahotas.imread("/home/jonathan/openplaques/blueness_images/1.jpg")
T = mahotas.otsu(v2)
imshow(v2 > T)

#Distance Transforms
dist = mahotas.distance(v2 > T)

Newcastle Maker Faire 2011

I'm writing this on the train back from Newcastle after attending this years Maker Faire. I've attended each Newcastle Maker Faire and it's been fantastic witnessing it grow each year. Many of the groups displaying their projects are veterans of previous Faires and it's inspiring seeing their projects develop from one year to the next. A growing faire means new groups and although some themes are repeated many groups have truly unique projects.

As last year I've put together a short video capturing some of the activity at this maker faire. It's difficult capturing more than a sliver of what makes this event great so I encourage you to click through to the project websites. I'll link to as many as I can below the video over the next few days but for the moment would encourage you to visit the official website.


I can't find a link for the roving wheelie bins but more information on the fire breathing dragon is available here.

I can't find a link for the first robot. The second robot was from mbed. The third robot was part of a very large exhibit but again I'm struggling to find a link. The fourth scene of robots were from robochallenge. The final ground based bot was from robosavvy.

The underwater bot was from underwater rov. The aerial photography using a model plane was done by Simon Clark. The rocketry was from Black Streak.

It was difficult keeping track of all the 3D printers so I'll just highlight two focusing specifically on 3D printers - bodgeitquick and emakershop.

The first automata was from Richard Broderick, the second from Philip Lowndes and I'm unsure about the third though I suspect Richard again.

The DIYBio presence was organised by Cathal Garvey and Brian Degger.

The standing wave flame tube was from Steve Mould. The wind up music disc was from the North of England Arduino Group organised by Mike Cook. I'm not sure who was responsible for the heart beat light sculpture. The interactive light table was built by Oli and the digital grafetti wall was built by the Jam Jar Collective and the musical tesla coils were from Brightarcs.

Read comments ...