Learning Python

Last September I participated in a one day full day basic Introduction to Python for Women workshop sponsored by PyStarPhilly. It was good exposure, but it didn't stick all that much.

Fortunately, they decided to hold the workshop again (details here), and this time made it even longer, starting on Friday night and all day on Saturday last weekend.

Friday night started with making sure the Python install on my machine I did last September was still OK, and that Notepad++ was up to date. And making friends again with my cmd window, which I also use to run the Perl scripts at work. Whee my cmd skills are rusty. But between work and this workshop, it's all surfacing from the deep recesses of my brain.

The rest of Friday can be summed up thusly: practice practice practice. The PyStar ladies have posted a series of exercises to practice the basics of Python on CodingBat. These exercises reinforce the concepts covered and force you to think through things.

Saturday morning began with completing Friday's CodingBat exercises and working through some Saturday morning review exercises. Practice practice practice!

A few things I learned while working on the exercises:

• spaces don't matter so much, although including spaces makes the code more readable
• punctuation matters!
• capitalization is noticed, so A =/= a, in other words, machines are very literal!

It is definitely sticking better this time. Between more coding stuff at work, plus Code Year, it's all starting to come together in my brain. I got farther with the exercises this time, and was able to reason through (or talk myself through) when I got stuck.

After a bit of warmup practice, we had a morning lecture. The goal of the lecture was for us to be able to write simple scripts on our own by the end of it. We reviewed the material in the morning exercises (equations, functions, etc.) and covered loops, while loops, input statements, dictionaries, more detailed functions, modules and packages. Also a brief description of libraries.

And then it was back to practice practice practice.

Another challenge I discovered: indentations. Where your indentations are impacts what your code returns, so you have to be careful what you embed under other things, versus what you want to do at the end. Often I wrote the code correctly, but indented it too far relating it to something it shouldn't be related to, producing no result, not even an error message. At one point I nearly created a recursive loop by indenting too far. Oops.

After lunch we had a few lightning talks, where people shared various code they're written for projects. It's encouraging to see what people have built; even more so that I can read and understand the code they wrote! The talks involved editing images (adding text), predictive puzzle solving (the Tower of Hanoi puzzle),   and a very cool project working with 7 to 13 year old girls and the Tropo simple API. We also looked at a very neat page of data visualization and graphical plotting options using Python at matplotlib.sourceforge.net.

The talks also include some more learn to code resources. I'll be posting the new ones I learned about to the catcode wiki resources page.

After the lightening talks, they gave us a project to work on (in addition to the CodingBat exercises) involving using Twitter's API to call recent tweets and trending topics. Challenging but so satisfying when you get it!

Why am I learning to code?

Why am I learning to code. I get asked this a fair amount these days. Why would a librarian want to learn to code? What good is it for my job?

I alluded to it in my first post on the catcode phenomenon: "Catalogers work with massive amounts of curated bibliographic data, and being able to manipulate it in new and different ways and in ever increasing amounts is key as we move forward into the bibliographic future and the world of linked data and the semantic web."

But that's a pretty generic statement. And doesn't explain much. I usually end up giving some examples of how coding would be (or already is) used in my current day to day work:

1. Editing of record sets prior to loading - beyond MarcEdit

All of our batch loads of MARC records for various collections go through a basic editing process. We have a standardized set of edits that all of our records go through to normalize certain elements and insert local-specific notes about access restrictions and our licenses. These edits are all combined in a script to facilitate the process. All sets are quickly run through the script as a first step of the loading process. Currently when something changes (like a change to MARC coding or the need to add something or add a conditional for another format such as images), I have to ask our systems folks to edit the script, I review it, we test it, and then it's put into regular use. If I could make the edits and test it myself the process would be much more efficient.

Often I have very large sets and potentially complicated edits that need to be made. MarcEdit has a script editor that allows me do make a series of basic conditional edits, but for more complicated things I'm still limited to asking our systems staff to write a script.

For example, our records from ProQuest for the online version of our dissertations come in with subject codes that are proprietary to ProQuest and their indexes. For these to be useful for our users for searching, they need to be mapped to valid Library of Congress Subject Headings. I can build a cross-walk in a spreadsheet for the mapping, but I need a script to run the records through a process to actually match on the codes in the records in the file and insert the LCSH heading.

2. Reports from our system (or from any large file of records)

The only mechanism for getting a report from our current ILS (Voyager) is via script. There's no reporting interface. Like with script editing, I have to pester systems folks to help me. And it can be a tedious process. I tell them what I need (being as clear and specific as possible within the bounds of how confusing MARC is and the myriad of exceptions/variations that exist). They then take my request, and turn the query into a script (interpreting things in the process), run it, and send it to me for review. If we're lucky, we get it right the first time. More often than not, there's errors, and exceptions, and quite a bit of back and forth before we get it right.

If I could gain some experience in coding, I could hopefully reduce the back and forth and write the exceptions and variations into my query request, essentially cutting out the middle man interpretation. Having a better understanding of what is possible to write into a query based on the data and how "clean" it is (or isn't) is invaluable.

3. Batch editing of records already in the system

Just because something is already cataloged and loaded doesn't mean you can ignore it. Records require maintenance. Rules change, headings change, tags change, new fields and subfields are added, etc. etc. MARC as a standard is changing regularly. Names are updated in authority files, subjects are created, collapsed, divided, etc. And we have to keep our data up to date so it's actually useful to our users and so relationships are maintained. All of this updating requires editing a very large batch of data. It starts with getting an effective/accurate report of things that need updating, and then telling the system what needs to change. A big thing that would be useful to articulate more clearly is conditional edits...add this field but only if these things are true. But if your instructions to the system isn't clear you end up with a big mess that you have to undo and try again, which is time consuming and most likely also problematic for user access until you clean things up (or undo). Once you have a better understanding of how to write the instructions, the number of messes decreases exponentially. And less mess is always a good thing when dealing with data that  potentially impacts user access.

4. Loading of records in bulk

There are set of profiles used for loading records into our ILS. Distilled down, they are a series of scripts dictating the modification and creation of bibliographic, holdings, and item records. The specifics of each load determine which profile is most appropriate. I don't have a good understanding of these profiles because my understanding of coding is still limited. I'm hoping learning coding will give me a better understanding so I may better advise which profile is more appropriate.

A good example of why this would be useful. A few months ago we tried to load a set of records into our system. Both the colleague in systems and myself being unfamiliar with the profiles, we picked the one we thought was correct. Well, given her unfamiliarity with MARC and library records, and my unfamiliarity with coding and in reading the profiles, we blundered and used the wrong one. We made a bit of a mess of things in our production catalog (hello error message displayed to the public) and had fix things after the load, once we figured out what we did wrong, of course. Our blunder created quite a bit of extra work for both of us.

5. System design / user interface design / etc.


We're currently in the process of redesigning our public interface to our catalog (our OPAC). This means dealing with indexes and the underlying data in different ways. Having a good solid understanding of the data means I can explain what data is useful and what our data can and cannot do. Writing functional specifications for the use/manipulation of the data so the coders can go to work. Having a solid understanding of how systems talk to each other and where data lives helps immensely. Learning to code helps me do all of that.



I think an overarching theme of all the above examples is communication. To be able to explain things to coders, and to understand the questions from coders, and explain things to catalogers, and non-catalogers, and effectively talk/explain my needs to the systems themselves, I need to have a better understanding of how it all works. Learning coding helps with that. It's like learning another language. If I can be even semi-fluent in coding, things will be much clearer for everyone involved in the conversation. Even if I don't become a full-fledged coder, the exposure I'm gaining from participating in Code Year and workshops on Python and attending project nights and talks about various coding languages and systems is already proving to be invaluable.

Library Day in the Life - Round 8, Day 5

This post is part of Library Day in the Life project. See the Library Day in the Life wiki for info on the project. I also participated previous rounds. You can find all my posts for this project by searching my blog's librarydayinthelife tag. You can also follow my Twitter posts @slmcdanold with the hashtag #libday8.

Friday, February 3, 2012 

I have much better luck making the bus this morning, but I forgot my travel mug full of coffee. I make a small detour between the bus stop and the library to buy coffee, again.

8:05am - Arrive at desk, my precious brew in hand. Login to the work network. This is followed by logging in to email, my work calendar, Voyager and Connexion, opening GTalk, opening a browser (Firefox at work, but Chrome at home), and logging in to various web programs (including Evernote), start TweetDeck.

Read email. Forward and respond to necessary messages. Catch up on news feeds, work email listservs, and library-related Twitter lists. Scan subjects and then delete mass numbers of listserv messages.

Check calendar to find out what's on today's schedule.

Walk over to supervisor's office to check in on a couple of backlog clean up projects we are working on: catching up on the Hebrew and South Asia materials.

Email off weekly timesheet report for staff to Library HR office.

Earlier this week the slate for the NASIG 2012 elections were announced. I'm up for secretary this year. We'll see what happens this time around as I've been on the slate before, but not elected. And apparently my opponent has friends that have volunteered to campaign for her! Oh dear. I better get on that as the ballots go out Feb. 27th. (hee!)

10:00am - Meeting to discuss preservation issues for serial back volumes.

Brief break of about 20 minutes between meetings to check email. 

11:00am - Meet with serials acquisitions librarian to sort out some issues with Hebrew serial titles that have been identified as our Hebraica cataloger makes headway on the backlog. Non-Roman scripts can cause confusion and things get linked to incorrect records, issues get skipped because of lack of proper title Romanization, etc. So we're clearing the confusion the best we can when we find the problems. Fortunately this takes all of 15 minutes to walk through and explain them and we can both move on to other things.

Since I have some unscheduled time that on my calendar looks like I'm occupied, I spend some time on that list of "web" location holdings that are problems. Turns out there's more than just the reserves issue that has shown up on our report. We also have random duplicate holdings, missing subfields, mysterious "web" holdings on print titles, and more. So it looks like we're going to have to go through all 269 titles one by one and by hand to fix them.

Noon - LUNCH.

12:30pm - I get some excellent news that the "new" workflow for individually purchased ebooks is up and running! We have it set up now that when an ebook is purchased, a brief record is put into our catalog so the payment record and invoice can be created. This payment record is suppressed. Once access is turned on, the payment record is unsuppressed, the URL from the vendor for our access is added to the bib, and the now unsuppressed brief bib is picked up by a daily report so it can be fully cataloged. Since we're adding the access URL to the brief bib, the public will have access immediately so if there's a delay in cataloging there's no impact on access. Now I just have to train myself to go retrieve the report every morning...

Send a few emails regarding the LITA/ALCTS Library Code Year Interest Group. We have approval from both ALCTS and LITA so we're legitimate!! YAY! ::Kermit flail:: Our ALCTS approval is conditional on some wordsmithing and revisions to the group purpose, but they approved us so we can move forward due to some impending deadlines. We're submitting our A/V and meeting room request for ALA Annual 2012 in Anaheim, CA! Watch the catcode wiki, catcode Twitter list, and the ALA Connect Code Year space for details. And don't be afraid to sign up (whether you're participating in Code Year or not) and join us in learning to code or improving our coding skills together.

Back to the spreadsheet of "web" location holdings. It's like I have 269 problem children who need individual attention. I do filter as much as possible, but I still have to check each and every one.

It's been a long day of staring at a screen full of MARC code and spreadsheets.


2:00pm - Meet with colleague to put together the charge for the subgroup to investigate the various metadata schema available that can accommodate preservation, descriptive, structural, and administrative elements for a wide variety of types of materials that may be included in the digital repository.


Give staff person the monthly report of items that are officially declared lost. 


3:45pm - Final email check. Clean out email listservs. Check Twitter and blog feeds. Schedule this post.


4:00pm - Shut things/programs down and log out from work network. Shut down (restart) computer. Leave work. 


I have to get out of here a bit early today as I'm attending an "Introduction to Python" workshop (more coding!) tonight and all day tomorrow sponsored by PhillyPUG / PyStar Philly. I need to get home to pack up my laptop and head out so I'm not late tonight (especially given my bus karma this week)! I did the intro to Python class last September, but it didn't really stick. So when the opportunity to take it again came up, I jumped at the opportunity. I'm hoping my Code Year work has helped so it sticks better this time.

Library Day in the Life - Round 8, Day 4

This post is part of Library Day in the Life project. See the Library Day in the Life wiki for info on the project. I also participated previous rounds. You can find all my posts for this project by searching my blog's librarydayinthelife tag. You can also follow my Twitter posts @slmcdanold with the hashtag #libday8.


Thursday, February 2, 2012 

I get to my desk around 8:55am, and yes, this is a bit late. My bus karma this week can only be described as terrible. Seriously epically bad bus karma. This morning I barely missed one (this week's prevailing theme...watching the bus go by as I'm about 1/2 a block from the stop), the second one didn't even pause, and the third (which I finally boarded) got caught behind a truck. EPICALLY BAD. Grumble...SEPTA...grumble.

Login to the work network. This is followed by logging in to email, my work calendar, Voyager and Connexion, opening GTalk, opening a browser (Firefox at work, but Chrome at home), and logging in to various web programs (including Evernote), start TweetDeck.

While things warm up, I retrieve my coffee travel thermos only to discover I didn't tighten the lid and it has dripped my precious brew all over the inside of my lunch bag. 

Can I go back to bed and start the day over now, please?

Read email. Forward and respond to necessary messages. Catch up on news feeds, work email listservs, and library-related Twitter lists. Scan subjects and then delete mass numbers of listserv messages.

Check calendar to find out what's on today's schedule.

9:15am - my phone rings. It's a random number. I pick up, the number hangs up. I do this twice before just letting it ring. The number calls repeatedly for the next 10 minutes, for a total of 7 more calls, each one ringing at least 4 times before hanging up. This is beyond annoying, to say the least.

I'd like to repeat my request to go back to bed now and start the day over.

That pesky reserves e-item multiple holdings records issue? It's still around. We're still working on it. That report from yesterday turned up a few other oddities that we have no idea what to do with such as personal copies with print and online components, and some that don't seem to be related to reserves at all.

10:10am - Stop by my supervisor's office to check in with her on a couple of things, including what we need to do in order to add our ebooks to our NewBooks+ stream. It highlights new acquisitions of books in the library (our media/videos are highlighted in a separate catalog), and we need to figure out how to fold in ebooks without overwhelming the system in terms of sheer number of records. Some of the MARC record sets we load for collections have over 50,000 records (some over 100,000!) so clearly we have to determine an effective filter for these extremely large sets.

10:30am - Department Heads. This meeting is a regular first Thursday of the month event. This month's topics include presentations on a special collections database from the museum folks, and a discussion about digital humanities. Oh, but if you get there early, there are treats! And coffee! I think the nice flaky pastry with icing on it helps to improve my morning, don't you agree?

Noon - Retrieval of MARC records for various ebook collections from vendors that post the files on websites. Some files I get an email alert, some are emailed to me as attachments, but a few vendors just post them without any notification options. I do have a "master" list of collections that are set up as ongoing subscriptions including how we get notified about updates in a Excel file shared with our electronic acquisitions folks so we can all keep it updated. But I have a tickler in my calendar and a running list in Evernote to remind me to go check the websites for specific subscriptions (mainly ongoing series, the online version of a standing order, or collections where they are still creating the MARC records) and pull down any new files. Usually there are only a few records in each update file, so I can quickly process and edit them and load them directly into our catalog. For the updates that I get via email or RSS notification, I process them as they arrive. I'm able to keep up currently, but in terms of scalability this is not the best method. I'm working on figuring out a workflow for these collections that get update files of MARC records that is a bit more streamlined and scalable. So far the sheer diversity (and seeming randomness) of how different vendors process and make updates available, and the variation in frequency for those updates, are making it difficult to have one workflow that encompasses them all.

12:30pm - LUNCH. The danger of eating lunch at your desk so you can catch up on stuff: colleagues stopping by with questions.

1pm - meeting of all the supervisors in technical services so our two new supervisors can get to know everyone.

3:00pm - Track down a colleague to talk about various workflow issues regarding upcoming work barcoding back volumes of serial titles. Send out email to all library staff announcing a set of MARC records for individual titles in a collection have been loaded and are available in the catalog. Successfully load

4:00pm - Successfully download update file or records, run it through the generic Perl script, make remaining edits, and load the completed records into our catalog. All without having to ask anyone else to take care of any part of it. It's very exciting to be able to complete the process without assistance! [Note: see yesterday's post explaining the excitement of being able to run sets through a script myself instead of asking our systems folks to do it for me.]

4:30pm - Check in with Hebraica cataloger about some serial titles in Hebrew we were working on yesterday. We're slowly working through the backlog so the titles are migrating over from her in a nice slow steady stream. It's not overwhelming, but we're making progress which is exactly what needs to happen.

4:45pm - Final email check. Clean out email listservs. Check Twitter and blog feeds. Schedule this post. 

5:00pm - Shut things/programs down and log out from work network. Shut down (restart) computer. Meet up with a friend to catch up over a couple of beverages. Yay for friend time!

Library Day in the Life - Round 8, Day 3

This post is part of Library Day in the Life project. See the Library Day in the Life wiki for info on the project. I also participated previous rounds. You can find all my posts for this project by searching my blog's librarydayinthelife tag. You can also follow my Twitter posts @slmcdanold with the hashtag #libday8.


Wednesday, February 1, 2012 

I get to my desk around 8:25am, login to the work network. This is followed by logging in to email, my work calendar, Voyager and Connexion, opening GTalk, opening a browser (Firefox at work, but Chrome at home), and logging in to various web programs (including Evernote), start TweetDeck.

Read email. Forward and respond to necessary messages. Catch up on news feeds, work email listservs, and library-related Twitter lists. Scan subjects and then delete mass numbers of listserv messages.

9:00am - Fill out survey from ALA on ALA Midwinter.

Pick up a piece of paper and find a sheet that looks suspiciously like a reimbursement form. Realize I forgot to turn in the paperwork and receipts to submit for reimbursement for ALA Midwinter. Sigh. ::headdesk:: Make the trek upstairs to turn things into the business office.

Check calendar to find out what's on today's schedule. The day is open. Whoa. What shall I do with myself today?

Morning: prepping and editing files of MARC records for bulk loads

Remember that weird character that showed up and caused records to error out when we tried to load them? I talked about it yesterday. It's not obvious when the character was introduced, so I'm going to review the files from each part of the process to find out when the random character string ({A0}) was inserted at the end of some of the subjects. This file goes through three scripts: the generic one to standardize things, a post generic script to map the subject codes to the LCSH equivalents, and finally a pre-load process to prep the records (adding data to generate holdings records upon loading, etc.). I have the individual output files for each step in that process to review and find when the character string showed up so we can troubleshoot.

I send off various requests for loading files of MARC records into the production instance of our catalog. Once I get the confirmation the records are loaded, I'll send out an announcement to all staff that title-level access is available. I also load smaller sets (less than 50 records) by hand into our system to speed things along.


10:10am - In order to help streamline the editing process for files of MARC records, I'm working with the systems colleague that normally runs the records through the scripts for me to get it set up so I can do it myself. This is very exciting. That means far fewer requests and less lag time for set editing. If I run new sets through the basic normalizing/standardizing script (FYI, it's in PERL, and all it does is standardize certain fields for local needs), then the only requests I need to make are for actual loads into our system! Exciting! Of course, this means installing software (ActivePerl community edition) on my desk and mapping me to various files, which takes a good 30 minutes. My colleague in systems is very patient with me, walking me through the steps (which I meticulously document in Notepad as we go through) and we have success! ::Kermit flail in celebration:: The process utilizes the DOS command window, and oh boy is my cmd vocabulary rusty. Yikes.


While we're chatting about running the MARC file editing scripts, I answer a few of her questions about load profiles and report frequency for a few ebook cataloging workflows we're tweaking. I discover in the course of the conversation that I was given SysAdmin access a few weeks ago but somehow I didn't get the message. SysAdmin installed and away we go! ::more Kermit flail:: Today is a good day!


Of course, the first attempt to do this on my own and I try to run the script on a file still in a .zip folder. Note to self: extract files first, THEN edit using Perl script. ::facepalm::


Noon - Try to figure out why we can't save edits to a bulk load profile when we change the order settings. I now have two chat windows open, one with my systems colleague and the other for a colleague that works in acquisitions to see if our inability to save a change to the orders piece of a bulk load profile is related to system permissions (I don't have permissions to edit in the acquisitions module).


12:40pm - LUNCH. Much needed. I think low blood sugar was starting to affect my brain (see attempt to edit file in .zip folder above).


While eating my soup, I respond to a message about putting together local specific training for MarcEdit and start to craft a possible outline, breaking things up into reasonable chunks from basic to really really complicated stuff. I don't know if I could do my job without MarcEdit anymore.


1:45pm - Next up is changing the way we process a specific MARC records subscription. I have to cancel one service, and set up a workflow for addressing changes by hand. This also means various requests to have records deleted from the catalog as well as a report of the deleted records so I can do the corresponding holdings maintenance. I put together the deletion request and report request and send it off to our systems department "ticket" system to add it to their queue. I also send off the message inquiring how to cancel the subscription to the MARC records files from OCLC WorldCat Collection Sets. 

I also get an email letting me know that the issue of multiple holdings records being created during the e-item creation process for reserves is much larger than just the handful I knew about yesterday. Our online resources should only have one holdings record with a "web" location and no other holdings locations or additional "web" holdings. Thus the creation of additional holdings records during the reserves e-item creation process is causing all sorts of display and processing issues. I request a report of all the bibliographic records affected so we can start cleaning up and start troubleshooting to find out what step in our procedure needs to be fixed to prevent the creation of multiple holdings records. Fortunately the report turns up less than 300, and it includes not just duplicate "web" holdings but also those where people are just using the wrong template to create "web" holdings. Completely manageable.



2:30pm - Help out our Hebraica cataloger with some serials questions and a cataloging module system question. The rest of my day is spent trying to track down errant issues, combing through backlogs in various places on our ground floor (aka the "basement") for materials in Hebrew, and trying to separate out titles that somehow got combined onto one record even though they are quite different. We gave up trying to figure out how and why it happened, and instead are just trying to fix it. Whew. I have stuff to talk to our serials acquisitions person about on Friday (he begged off talking about it tomorrow, so Friday it is).


4:15pm - Final email check. Clean out email listservs. Check Twitter and blog feeds. Schedule this post.


4:30pm - Shut things/programs down and log out from work network. Shut down (restart) computer. Drop of reimbursement form on my way out.