All of the WordPress themes that I work on for iThemes are managed as Git repositories. Recently, we moved past the 100 repositories mark. That’s a lot of repositories to manage, and unfortunately, too many of those repositories contain duplicated information.

Later on, I might delve into how we use Git to manage our theme repos. For today, however, I’d like to focus on how I quickly and easily pushed up changes to more than a dozen repos in a single, albeit long, Bash command.

I had finished making updates to 16 Flexx repos, and I needed to push all of those changes up. Since I had multiple working repos in that folder, I was lucky that each of these repos began with the text “Flexx”. Also, since they are all part of the same series and need to keep the same version number, that simplified the tagging as all could be tagged as 2.5.0.

Given this information, I simply ran the following command from the directory that contained all the repository directories:

There’s a lot going on here, so I’ll break it up and explain what I’m doing.

This is basically a compound command. Bash’s for command is being used to step through the results of ls|grep Flexx. Each result will be stored to the variable i which can be referred to with $i.

In other words, I’m going to loop through each directory that contains the text “Flexx” (technically, it isn’t limited to just directories, but I don’t have any files containing “Flexx”, so I’m safe). For each iteration of the loop, the variable $i will contain the name of the folder I’m interested in.

The most important bit here is do. do simply begins the functional part of the loop and could be followed by any command that I wanted to start the loop with.

The echo command isn’t technically necessary. I simply have it there so that I can better determine what output belongs to which repository.

This is a very important command. It changes the directory to the repository I wish to run commands on.

This is the command that actually does what I want to do in each repo directory. This adds all the modified files, commits them with a message of “2.5.0″, pushes the changes to the remote repository, creates a new tag of 2.5.0, and pushes up the new tag.

I have all of these commands chained together with && so that if one command fails, the other ones won’t run. This prevents tagging in the event that the repository couldn’t be committed or pushed.

It’s a small command but necessary. This switches back to the main directory that holds all the repos, thus returning us back to a state that the next loop iteration can work with.

As with the other echo command, this simply allows me to keep track of what is going on more easily.

The done command finishes out the loop iteration.

I hope that by sharing this you can gain a bit of insight into how I work with repositories while also learning more about how you can do things with Bash.

For about eight years now, I’ve been hapily using Crimson Editor. At its simplest, Crimson Editor is nothing more than a fancy Notepad; however, it is so much more than that. While it doesn’t have all sorts of fancy tools that some editors have, it does support customizable syntax highlighting, customizable colors, line number, visible whitespace characters, basic smart indenting, a file/directory navigation pane, project support, and a MDI interface. Those features aren’t really the reason why I use it as almost all programming editors support such features. I use it because it gives me tools that help me program without doing things that hinder my productivity.

Two relatively-small features that most people would miss but make working with Crimson Editor a pleasure are as follows:

• Built-in macro ability with quick access keys – I have a number of pre-defined macros that do all sorts of tasks that I find take too much time without. For example, I have macros that duplicate the line that I’m currenlty on, replace all the white-space characters at the front of the line with a single tab (this makes reformatting files with spaces for tabs much quicker), comment out or remove he comment from the current line, and so on.The addition of a quick-record button makes doing a number of repeat tasks extremely easy. I once had to remove every third line from a document that was 500 lines long. It would take too long to do it line-by-line and wasn’t worth the time required to crank out a script to do it for me. Using the quick macro ability, I created a macro and had all those lines removed in less than a minute.
• Regular expressions in search and replace – A number of programming tools are starting to add support for this, but Crimson Editor was one of the first ones that I encountered with this ability. Even many of the newer editors that I have tried fail at reproducing the ease that Crimson Editor has when it comes to this extremely helpful ability. You can even perform the replace command on all the currently open files at the same time.
• Remembers what I was doing last time I had the editor open – This is a slick feature that some of the newer editors are starting to pick up on. When I’m done for the day, I close everything out. When I come in the next day and open up Crimson Editor, it opens up all the files that I had open last time. Heck, it even puts the cursor where it was in all the files that I had opened previously.

I think that editors these days try too hard. Rather than providing extremely robust syntax highlighting and a slick/customizable editor, they build in all these tools that are supposed to make my life easier. Unfortunately, most of these tools are touted as the “features” of the editor and cannot be disabled. Even more upsetting is that many of these tools actually decrease my productivity.

There are a number of trends in the current generation of editors that are coming these days that I really don’t like. Once again, it’s not that I think everything should be changed to accommodate me, I just think that the developers of these editors need to understand that not everyone wants feature X thus features that affect how editing works should be have the option of being disabled/enabled. I’ll pick on E-Text Editor as it is so close to being great, but it has some of these failings as listed below:

• Constantly adding characters that I didn’t type – This is a major point of frustration for me. So many of the modern editors that I have tried out want to “help” me write my programs. In their attempt to do so, they make mistakes and introduce syntax errors into my code. Those mistakes go by unnoticed by me as I can type out most of my code without looking and often am not looking since I’m looking at a terminal window, database output, or other information source while programming.For those who don’t know what I’m talking about, take E-Text Editor as a good example of this behavior. Whenever I create an open parenthesis, it automatically creates a closed parenthesis. This seems like a nice way to prevent from forgetting a close parenthesis, but what if you are putting together something that requires an open parenthesis but not a close (such as regular expressions or adding new conditional checks inside an existing conditional)? Suddenly you have new characters that you may forget were created peppered around in your code that will create syntax errors. Frankly, I produce fewer syntax errors because of forgetting to close a paren than I did because of unnecessary close parens added by my editor while using E.
• Smart tabs that get too smart for their own good – I enjoy smart tabs. I like automatically having my next line indented after adding an open curly brace. I don’t like it when my editor continuously tries to force what it believes to be proper tabbing after I finish typing out a line. If I want to break out of the standard tab flow for whatever reason, that is my business. E-Text Editor, like so many modern text editors, refuses to allow me to do this. I say refuse, but it’s not always a 100% thing, which drives me nuts more than anything. Every so often, if I start a new line outside the common tab flow in E-Text Editor, when I hit enter, the previous line gets auto-tabbed and my new line is tabbed to match. No! I do not want that line there. If I go back and correct the auto-correction, as soon as I arrow out of the line, E moves it right back to where it was before. If I explicitly go outside of the “smart” tabbing, my editor should no longer try to force its rules on me.
• Ability to remember some things but not others – As mentioned before, I love how Crimson Editor remembers to open up all the files as I had them last time it was run. E-Text Editor, as well as many other newer editors, also have this feature. However, I feel that they aren’t doing enough. E-Text Editor implements code-folding, which is a very nice tool BTW. However, if you have lots of files open with many areas of code folded and you close the editor, when E reopens, all the files won’t be as you had left them; rather, they all open up but all the code folding is now gone. Not restoring the code folding state seems to be a major oversight to me as I want to come back to not just the same files I had yesterday but to the exact same state that my editor was in when I closed it. I really don’t want to spend the first part of my day refolding all my files so that I can get back to work.

I suppose to some, these are not earth-shattering problems. I thought I could deal with them too and used E-Text Editor for the full 30 days of the trial. I fully expected to have my programming worldview changed and to fall in love with E. Instead, I was just as frustrated by the problems as I was on day one. I never got passed how I felt like some parts of E were just as annoying as Microsoft Word when it autocorrects things. At least in Word, you could disable autocorrect. I wouldn’t be surprised if someone tells me to just turn off smart tabs in E. It’s not that I don’t like the smart tabs, it’s that I don’t like how E’s implementation of smart tabs doesn’t let me modify the text as I want it. I want it to auto-tab as I type but not enforce its will when I’m manually formating.

I suppose the crux of the problem I have with E, and most other next-gen editors, is that the features that are supposed to make me feel powerful and productive all too often times make me feel controlled and frustrated. Who knows… E is still new. I very well may be a convert if it becomes a bit more flexable.

Web applications developers such as myself often have a hard time keeping up with everything that’s going on. There’s always some new programming or scripting language, new standards, new browsers, new technologies, new paradigms, new social networks, and on and on. Every day, something new happens. The end result of all of this always ends up being very similar: people demand faster applications that deal with ever-increasing amounts of data which end up putting massive stress on the server architecture.

As we toil to improve the performance of the applications and their snappy response times, it becomes easy to forget about how we can tweak settings on a low level to provide massive speed improvements on the front-end. For example, most people don’t know that you can configure MySQL to take better advantage of the resources that the server has available.

By default, MySQL is configured to consume a relatively limited amount of memory resources. Start giving MySQL more memory to work with, and your application’s performance can improve greatly.

Unfortunately, I’m not a MySQL configuration guru. I’ve never read through the source code or any low-level technical documents about how it functions. So most of the configurations options are a complete mystery to me, and I don’t know if changing a specific option could benefit me at all.

Fortunately, I don’t have to be a MySQL configuration guru to get better performance. There is an amazing tool created by Matthew Montgomery called the MySQL performance tuning primer script.

Matthew Montgomery’s script does an amazing job of giving me information about how my MySQL configuration may be inappropriate for the queries being executed and provides information about what variables to modify in order to better optimize my configuration. For example, the “TEMP TABLES” section of the output tells me the following:

TEMP TABLES
Current max_heap_table_size = 512 M
Current tmp_table_size = 512 M
Of 1684 temp tables, 10% were created on disk
Effective in-memory tmp_table_size is limited to max_heap_table_size.
Created disk tmp tables ratio seems fine

Previously, I had my max_heap_table_size and tmp_table_size variables set to default settings and had up to 60% of my temp tables created on disk. As much as possible, temp tables should be created in memory to reduce the performance penalty of using the disk for random access to such data rather than memory which is much faster.

Now this doesn’t mean that you should set your variables to the same settings as I have. The server that this is running on is dedicated to running a dispatching system that has many tables containing hundreds of thousands of rows each. The value of the script is that it will help you tailor your settings to best use your hardware without having MySQL consume too many resources that other processes (such as you web server) could use.

The script is provided as a standard Linux shell script. Download the script into a folder of your choosing and run “sh tuning-primer.sh“.

Good luck with your MySQL optimization. I hope that you get as great of results as I have. Keep in mind that if your table structures and queries are inefficient to begin with, that configuration optimization can only do so much to improve the performance.

Further Reading

• Tuning LAMP systems, Part 3: Tuning your MySQL server
  This is a great read that should help most novice database administrators understand some of the basics about tuning their database’s performance.
• MySQL Performance Blog
  A blog by the authors of High Performance MySQL (also highly recommended). This blog is updated frequently with great tips that will help you develop higher-performance apps.

I work for a company that has a proprietary CMS software package written in Perl. We have a number of servers that run this code, and everything has been fine for a number of years and many different versions and customizations.  A few months back, something changed. One of our servers started producing completely empty files for all uploads. This affected all of our code, all of our versions, and every site.

After some time debugging and testing, I finally found the problem. Apparently some code was updated, either Perl itself or one of its packages, and that caused my CGI object to be recylced before the upload code ran. When the CGI object gets recylced, all the file handles are closed resulting in reading and saving an empty file.

The solution was deceptively simple. All I had to do was store the CGI object in a persistent variable that has scope throughout the end of the program execution.

This time I have a plugin for the masses. WP Easy Uploader will make any WordPress user’s life easier. No longer will you have to rely on FTP to upload new plugins, themes, or random files. Now you can take care of those tasks quickly and easily from the admin screen of WordPress.

For full details, check out WP Easy Uploader’s release page.

Starting a little over a month ago, I started submitting plugins to WordPress.org’s Plugin Directory. Overall, the system is great for developers since it offers Subversion and enables the automatic plugin upgrades that the latest WordPress versions support. However, it has some major problems that causes developers like me some headaches.

The instructions on how the request for adding new plugins works has the following information:

1. Sign up.
2. Within some vaguely defined amount of time, some one will approve your request.
3. You’ll then have access to a Subversion Repository where you’ll store your plugin.
4. Once you put your plugin (and a readme file!) in that repository, it will shortly be automatically entered into the plugins browser.

The “vaguely defined amount of time” isn’t defined at all. I’m sure that there are some people that handle new plugin approvals on their spare time and set up the subversion for the new projects, so I understand how there can’t be definitive times for new plugin approvals. However, I think the language here needs to clarify the details of how the approval process works and what type of feedback will be sent when the approval process is complete.

I was pulling out my hair when trying to submit my first plugin. I put in the request and then checked back a few hours later, nothing had changed. I checked a day later, not approved. Finally, after three days, the site said that my plugin was approved. I eagerly dug into the documentation on how to add the plugin to Subversion and started the submission process. When it came time to check in the repository (this is how you submit your new version), I received the following message:

Error: Commit failed (details follow):
Error: CHECKOUT of ‘/!svn/ver/52171/current-date-time-widget/trunk’: 403 Forbidden (http://svn.wp-plugins.org)

Forbidden? There must be an easy fix for this problem. I started searching around Forums, and found a large number of people were having the same problem. No one had any help for these people and there weren’t any official responses. Everyone just guessed that some random accounts just aren’t set up correctly and that there isn’t any way to fix it or to get help. After making official help requests and sending an email off to Matt Mullenweg with no response to either, I figured that they were probably right.

A few days after seeing that my submission request had been approved, I received an email telling me that my request was approved and that my Subversion would be ready in about an hour. I was extremely happy to get this email but was also very confused. If I had known that I just had to wait for an email, I would have just waited patiently. Instead, I wasted hours of time searching the forum, requesting official help, and doing some trial-and-error testing.

So what do I want to do about this? I’d like for the documentation to be improved. Sure, it’s kind of cute, but it doesn’t help a developer know what to do when they run into the problem I did. The documentation should read:

1. Sign up.
2. Approval of your request can take anywhere from a few hours to more than a week.
3. When your request is approved, you will be sent an email notifying you of the request approval with instructions for submitting your plugin to the Subversion Repository.
4. Once you put your plugin (and a readme file!) in that repository, it will shortly be automatically entered into the plugins browser.

The change is simple, but it would prevent a lot of frustration that may cause promising developers to ignore the repository due to their frustration.

Just last night, I found a new irritation with the plugin submission system. Since approval takes so long, I thought I’d go ahead and put in requests for some plugins that I’m working on so that when they are ready to be released, their spot is ready and waiting. I released the WP Developer Assistant late last night and was dismayed to see that my plugin wasn’t listed on the new plugins page. I went to bed thinking that I would see my plugin listed there today. For those that don’t know, getting a plugin listed on that page is crucial since you get a lot of exposure and can build an initial user base quickly. To my disappointment, I didn’t find the plugin on the new page today either. After a “what if” thought, I went back a couple pages and found my plugin listed on the third page. Apparently, the plugins are ordered in that list based upon when the plugin request was approved, not when the first version is loaded.

Due to my desire to release the plugin as soon as it was ready rather than waiting for request approval before I can release, my plugin has lost it’s best chance to get good initial exposure. I’m not sure how the back-end of the system is structured, but it would make sense to me that plugins should be listed in order of their initial version release rather than by their approval date.

Please don’t misunderstand me, I love the system. I just think that it has some issues that could be simple to fix and would improve the experience greatly. I’m still sad that my plugin that I’m really excited about didn’t get a chance to sit on the front page for a few days. Oh well, maybe next time.

Time to release another public plugin into the wild. This plugin is a bit different in that it has nothing to do with how your site looks or operates. As a matter of fact, no one will even know that you have it, but you can rest well knowing that it is just a click away in the admin menu. For you see, WP Developer Assistant is a plugin for developers.

Now some of you may wonder what a developer needs a plugin for. Well, there are lots of reasons. The main one being that it would be a lot nicer if WordPress had a few extra tools that would make a developer’s life a lot easier. For example, being able to run any needed query from inside WordPress, being able to show PHP errors without turning them on for everyone, or uploading files and extracting archives anywhere in the site would be great tools to have, but WordPress doesn’t have anything like that… Until now!

If you are a WordPress developer, you need this plugin. You can find the full details on the WP Developer Assistant page.