OSSMichigan.org - Not quite a planet but a state.

August 27, 2008

Mark Ramm

TurboGears 1.0.6 released

The TurboGears community is pleased to announced that TurboGears 1.0.6 has been released. This version is a maintenance release that fixes bugs & glitches found since the last two months.

We are really active at the moment working on the 1.1 branch which should enter beta tomorrow. The 1.1 release will be fully retro-compatible with the 1.0 branch but will propose important new things:

  • SQLAlchemy as the default ORM
  • Genshi has the default templating engine
  • A new test system that is forward compatible with branch 1.5 & 2.0 (this is an important step toward migrating old applications written in 1.0 to 2.0 without breaking everything in the process)
  • Drop the use of RuleDispatch and use Peak Rules instead (no more C compilation)
  • Enhanced mod_wsgi compatibility to avoid past hacks that we had to do to deploy on Apache

Once 1.1 is out has a stable version we will encourage all 1.x users to go with this new version and to update their unit tests by following the TestMigration procedure.

A big work is undergoing also in the 1.5 branch to bring you CherryPy 3.1 support and further move TG into the WSGI realm.

Enjoy this new release and give feed-back in our mailing list!

Florent Aide.

by Florent Aide at August 27, 2008 06:45 AM

Jorge Castro

Use UNIX or Die ...

Longtime sysadmin guy Ben Rockwood is apparently starting a podcast for sysadmins.

Ben and Cuddletech have been around /forever/ ... I remember his Enlightment pages being a boon for me in the late 90's when I was on my E kick. (Remember kids, everyone goes through an E phase.)

by jorge castro at August 27, 2008 02:03 AM

August 25, 2008

Jorge Castro

Help test Openoffice 3.0

Chris "Chain Man" Cheney has put out a call for testing for OpenOffice 3.0.

Chris was just mentioning to me the other day how he doesn't have nearly enough bugs assigned to him and how he would love to have the community flood him with 3.0 bugs. I was shocked that he would be such a sacrificial lamb, but history is full of daring men and women who have taken one for the team.

Note: previous paragraph is a complete fabrication.

by jorge castro at August 25, 2008 07:41 PM

Rick Harding

amazing how seeing one post can fire off a whole train of thought

I ran into this post talking about how he's always uploading files for people to see. It really hit home. In the past few days I've been uploading some custom tweaks to the Gwibber ui along with my irssi config for people to see. I launch my ftp client, upload the files, wheee. The idea in that blog post is that you could just have it things prewired to just launch any file you wanted and then return a link to that file.

That's cool and all, but I have a couple places I upload things. I have a different spot for work/home things. I also was reviewing an article on Amazon Web Services for PyMag and I got thinking, how could would it be to have the script be able to upload to S3 as a publicly available flie, and then get a link back to it.

So I started working on a new app I'm calling sendoff. It's a lot more complicated than that bash script, but it allows you to define multiple server locations, have sane defaults, and there's more to come. A lot more. I need to figure out how to display progress information, get hooks for S3 and FTP uploads, etc. Right now it only supports scp via the paramiko library (very cool btw).

It's already very cool though. I can now upload by default to my home server with

sendoff filename.jpg

and I can do it to work with

sendoff -h work filename.jpg filename2.jpg

So far it's been fun. My first time creating an easy_install-able package from something. So take a peek at it if you want, let me know if you see anything in particular. It's a lot of firsts in there so I know it's a mess and I'm working on cleaning it up bit by bit. Looks like I've found a better project for MiDeveloper meetings for a while.

by nospam@example.com (Rick Harding) at August 25, 2008 12:39 AM

August 22, 2008

Andrew Turner

Visualizing Restaurant Searching

24 hours of Urbanspoon on the iPhone One of the simplest, most useful, and well executed applications on the iPhone is UrbanSpoon’s free restaurant finder. Open the application, it geolocates you, give the iPhone and shake and three Slot machine style selectors spin around and randomly choose a restaurant nearby. You can then even lock in specifics such as location, cuisine, or price range and shake to give more suggestions.

However, what’s particularly neat about the application is that UrbanSpoon has been recording these “shakes” by location (inherent in the API call obviously) and created a great visualization showing the locations around the US over a day of “shaking”.

The heatmap essentially shows the evolving enquiry of people looking for a place to eat. There isn’t an actual time display or timeline slider to investigate - but I imagine there are interesting trends during meals, and particularly after normal eating times when people don’t have a plans on where to eat. In addition, a timezone lag that would show shaking progressing east to west.

Context mining of mobile devices, combined with geographic location - and especially via inferred geographic information instead of directly volunteered information can yield interesting trends on ambient behaviors. Imagine if UrbanSpoon could also collect the number of people in the group by detecting other repeatedly seen nearby bluetooth/wifi devices, previous meals of the day, and the ultimate destination and distance to the chosen restaurant.

by Andrew at August 22, 2008 06:38 PM

August 21, 2008

Jorge Castro

August 20, 2008

Jay "jwren" Wren

New Server and My Worthless Linux Account Management

I got a new server for my home, more to come on that later. Right now I want to talk about my worthless account management practices. You see, a long time ago (4 years) in a land far far away (2 counties), I was a system administrator. I knew all the best practices. I could plan a server migration with zero downtime or if some DT was required  could minimize it. I could keep my servers up with 3-4 9’s (99.99?%).

I think I’ll compare my situation to that of the super clean janitor who invites you into their home only to reveal that they live in squalor. My home server works, but it is a mess. So it is wholly appropriate that when it came time to replace my aging server with 3-4 years of Linux cruft (initially installed with Breezy Badger and then upgraded) that I throw caution to the wind and just install Linux and see what kind of problems I run into.

Well, the first problem I run into is that my baby cannot watch her Baby Signing Time DVD rips from the server — I know ripping DVDs is a violation of the DMCA, but Jeff Atwood says it is OK, so if you are an MPAA lawyer, please see Jeff, he is a top-10 blogger and must have way more money than I do — the XBMC wouldn’t work. Well DUH!  The only account on the new system is mine. I hadn’t created any XBMC account for the samba share, nor had I set its samba password.

Since I’m sure many other people might reinstall and not upgrade, but would like to not have to reconfigure everything, this is my little migration guide. I’ll assume that you have a backup or your old hard disk lying around and that you are capable of getting at the needed files in some way. In my case, my old root file system is mounted /mnt, so when you see /mnt paths in these commands, just remember that is my old disk.

Jay’s Stupid Guide to Account Migration

First, it is important to keep in mind that Ubuntu, Debian and AFAIK every Linux distribution these days creates necessary system user accounts for you on demand. e.g. I didn’t have a bind account and group on my system until I installed the bind9 package. This means the the uid/username map will most certainly be different between your old system and your new system. I’m OK with this. I want a new system. I just want some old user accounts.

Visual Inspection

$ cat /etc/passwd

….

jrwren:x:1000:1000:Jay R. Wren,,,:/home/jrwren:/bin/bash

janice:x:1001:1001::/home/janice:/bin/bash

sbak:x:1003:1003::/home/sbak:/bin/bash

….

Cool, so there is my account, and there are a few others. One for my wife, one for a great friend, and I left out a few more. But there aren’t too many, so if I had to manually do this, I could do that. If I really had to copy and paste lines in passwd and shadow I could. But I won’t.

Let Them Login

Use your eyes and know your regex. In my case the uids for all of the accounts I wanted moved were 1001-1007, so I confirmed that my regex was correct and then just appended via pipe using tee.

$ grep x:100[1234567] /mnt/etc/passwd | sudo tee -a /etc/passwd

I could now run the “id” command for these new users and it would work!  Neato.

But I’d like them to be able to login with their old password in case they were at some place without their ssh public key, so I set out to do the same thing with shadow. Shadow is keyed by username, not uid, so we just have to map back to that.

$ grep x:100[1234567] /mnt/etc/passwd | awk -F: ‘{print $1}’ | sudo grep -f - /mnt/etc/shadow | sudo tee -a /etc/shadow

Cool, now people can login.

Power In Numbers and Groups

If we look at the stat(3) of a file from the old system as owned by one of these migrated users now, then the uid maps to the correct name but the group is incorrect. We need to do the same thing for groups.

$ grep x:100[1234567] /mnt/etc/passwd | awk -F: ‘{print “x:”$4}’ | grep -f - /mnt/etc/group | sudo tee -a /etc/group

So we map a little differently and we construct our fgrep regex using awk, but its mostly similar to the shadow case.

Samba

Regular user accounts are all neat and good, but the whole point was XBMC which connects via samba share. This is where my sysadmin knowledge is starting to show its age. The last time I migrated samba accounts the accounts were in an smbpasswd file and could be moved very much like the shadow case above. These days samba stores this information in a trivial database known as a tdb. http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/msdfs.html (No idea why an html file named msdfs actually contains tdb information instead of DFS information.)

I had to search the docs and google a bit before I found this post: http://fixunix.com/samba/246816-samba-where-samba-store-users-passwords.html  which lead me to the pdbedit command. Samba has changed since I last used it. http://us3.samba.org/samba/docs/man/Samba-HOWTO-Collection/passdb.html

Unfortunately, I didn’t have tdb password entries for all these user accounts, actually I had ‘em only for two. No big deal since the easiest path is to go through the old smbpasswd file format to get here.

First thing, I found that pdbedit doesn’t like read only file systems, so copy your pdb local (did I mention my old disk is mounted read only at /mnt?)

$ cp /mnt/var/lib/samba/passwd.tdb oldpasswd.tdb

Now we need that old smbpasswd file format

$ sudo pdbedit -Lw -b tdbsam:oldpasswd.tdb  > smbpasswd

Then I want to filter it for only the accounts I care about.

$ grep x:100[1234567] /mnt/etc/passwd | awk -F: ‘{print $1}’| grep -f - smbpasswd > smbpasswd.stripped

Then (thanks to abartlet #samba on freenode for guiding me) merge these into the new systems passwd.tdb

$ sudo pdbedit -i smbpasswd:smbpasswd.stripped

Now you should have functioning samba passwords.

If someone reading this is thinking, oh this is WAY easier in windows, could you please link me to the step by step process like this? I’ve always wanted to use it and I could never get it done.

by jrwren at August 20, 2008 04:10 AM

Jorge Castro

Get your hot gwibber action here ...

Screenshotgwibber_2Ok, so when Ryan Paul (from Ars Technica) started working on gwibber as a twitter client I was mildly interested. It wasn't until identi.ca was launched that I moved wholesale and needed a good client. (Twhirl and Spaz are good too but the Linux Adobe AIR alpha is holding them back).

gwibber was kind of Ryan's playground application where he could mess around with things like webkit and GTK stuff. However, people wanted an identi.ca client, and started using it. Ryan wanted to do the packaging himself to learn it (and hopefully write it up for open.ended) so we specifically avoided packaging it. Well, it became more and more popular, and it was time to make packages, it was already in bzr and on launchpad, so it would be heresy to not leverage the awesome power of Personal Package Archives.

This evening I registered the gwibber-team and set up a PPA. Instructions are here. Note that it depends on the webkit PPA, so you need both sets of sources. gwibber uses webkit to make the cool layout you see in the screenshot, complete with reflections and rounded corners!

Before anyone asks, no, I won't be putting these up for Intrepid proper, gwibber is still fast moving and needs work. Branches are popping up all over the place and people are Doing Great Things(tm), but it's just too fast moving to maintain in the distro, which is why we have PPAs! With webkit coming in a future version of GNOME it should start to settle down. You'll find my packaging branch at the previous URL, if you want to jump in and help out then let me know!

You can follow me on identi.ca using gwibber, I am @jorge, and you can follow Ryan too, @segphault. Come join the other bunches of Ubuntu users on identi.ca!

Installation notes for our openSUSE friends are here. If someone wants to throw these into Debian post-freeze then please let me know, and of course feel free to steal my packages!

by jorge castro at August 20, 2008 03:05 AM

August 19, 2008

Jorge Castro

Feeding the Harvesting Machine

While we were at Debconf Jono and I tweaked the 5-a-day page to be simpler and less confusing. One list of possible targets we added is the list of unlinked upstream bugs. These are lists of bugs where people have pasted in bug reports in comments but have NOT linked to it via Launchpad's "Also affects project..." feature.

Why linking bugs upstream is important

Just mentioning a bug report that is upstream is only one aspect - sure it's useful, but linking it allows us to query Launchpad for these bugs, and more importantly track these bugs programmatically so they get on the right person's radar to fix it. Instructions for doing this are here. Now, let me show you an example on why this is important.

Example bug

Bernhard Schmidt reported this bug on rdesktop on February 8th. Basically, he wanted rdesktop to be compiled with IPV6 support. Bernhard also opened up a bug in Debian. Three days ago Laszlo Boszormenyi reported that it was fixed in Debian with the upload of rdesktop 1.6.0-2.

Ok so now what? The bug just sits there right? Well, that depends. As it happens, I was doing my 5-a-day and stumbled across this bug - and I thought "This bug is fixed in Debian already, it should be on someone's hit list to fix, right?" Wrong. Or am I? No way to tell for sure, and short of Bernhard chasing someone down or having someone else who can actually fix it find it isn't very efficient. If only there was a list of easy, low hanging fruit ... like say Harvest!

I linked the bug to Debian bug in Launchpad. The next time the bug watch is updated Harvest will find it, and it will show up in Harvest automatically. Now as people trudge through Harvest the bug is listed as a possible fix and they can fix it.

Oiling the machine

So, what needs to happen for this to be as efficient as possible?

  • Bug reporters - Do continue to report bugs upstream like Bernhard did in this case. Well done Bernhard! Instructions for filing bugs in upstream trackers are here.
  • Bug reporters and triagers - Don't just throw a URL in the comment box, link it via Launchpad (instructions).
  • Bugsquad / Experienced Bug Triagers - encourage people to link bugs during Hug Days.
  • Ubuntu Developers - Keep monitoring Harvest for easy "sync-from-Debian-or-upstream" bugs.

Lessons Learned

As you can see, linking bugs upstream helps get bugs moving in the process, and eventually end up in things like Harvest or the "bugs fixed elsewhere" pages. People write tools and scripts to find these bugs, especially if they're resolved upstream, which makes it easier for people to find so that the fixes can be shipped to users, which is what this is all about!

Now let me put my Jordan Mantha hat on: "But Jorge, getting it on Harvest is one thing, that's only half the solution. Getting someone to find it and fix it on the other hand ..." Yes, absolutely. If you think as this entire Harvest workflow as one big machine, then upstream linkages are feeding the machine, but at the end of the day it needs to spit out bushels out the back.

So we need developers to fix these bugs - which is why we have things like Ubuntu Developer Week, MOTU mentorship, sponsorship, Harvest, etc to grow the developer community. This is a tough area to grow, but we are making progress, and tools like Harvest are making things easier I think, in the meantime, keeping the machine well fed is still better than a bug sitting in Launchpad all by itself ...

by jorge castro at August 19, 2008 05:17 PM

August 13, 2008

Andrew Turner

Open-Source in Defense

One of the more interesting presentations and discussions at BarCamp.mil was the Department of Defense CIO thoughts on a future publicized guidance on the use and promotion of open-source software for defense contracts.

Open-source in the DoD isn’t new. In fact, there have been government reports that call for the DoD to issue an official strategy for utilizing open-source. The issue is particularly unclear in recent light of the US Government being declared a sovereign entity outside of copyright law.

The objective of the prospective official guidance would be to outline both the benefits of using open-source in defense as well as provide understanding on the effects to contractors and bids. It is vital that contracts are clear on the legal implications and responsibilities of providers.

So far, this has meant that solution providers are not necessarily willing to wade through the uncharted waters and instead deliver proprietary software. In addition, there was no impetus to use open-source, since by shipping proprietary software the effect is to lock-in the provider for decades.

In addition, there needs to be a defined mechanism for how accepted open-source software that comes inside various government offices, especially defense and intelligence organizations, be re-released to the community. Currently I’ve seen a number of open-source projects taken into agencies and essentially forked since the code will never be able to be released due to concerns of potential security leaks in the code itself.

In the open-source world, a government supported promotion of its use would have dramatic effects. Looking at the current state of commercial company support for projects such as Apache, Linux, Gnome, OSGeo and more demonstrate that there is clear benefit to be gained. If the government then pushes open-source there would a huge upsurge in the support of projects and communities.

Here’s to hoping the lobbyist groups don’t head this off before it makes it to the light of day.

by Andrew at August 13, 2008 06:45 PM