Introduction
In planning and launching new Web sites, the emphasis is too often firmly placed on the period leading up to launch, with little regard to the critical post-launch phase. Much like an elegant ocean liner heading off on its maiden voyage with the champagne still wet on the prow, nobody is concerned with the day-to-day realities of the crew as they are left to ensure continued smooth sailing and navigate around icebergs.
This paper brings to light some of the strategic and practical issues that are likely to emerge in this often overlooked and under-considered period in most on-line projects, and attempts to debunk some of the unrealistic expectations associated with major redesigns. Focusing on three large-scale, U.S.- and U.K-based organizations, the Museum of Modern Art (MoMA, http://www.moma.org), the National Gallery, London (NG, http://www.nationalgallery.org.uk), and the San Francisco Museum of Modern Art (SFMOMA, http://www.sfmoma.org), we consider universal themes that have emerged from our shared experience of deploying complex digital projects, specifically exploring the issues we faced after our sites went live.
Fig 1: MoMA's homepage, http://www.moma.org
Fig 2: NG's homepage (left) http://www.nationalgallery.org.uk and Paintings page (right) http://nationalgallery.org.uk/paintings/
Fig 3: SFMOMA's homepage, http://www.sfmoma.org
These key issues range from the unexpected impact of a redesign on an organization's strategic planning to the ongoing difficulties of measuring success. We have provided examples of the most significant challenges we faced, shared the pragmatic decisions we made, and acknowledged the many unresolved issues we face.
This paper is aimed at those leading Web site redesign projects, and is a follow-up to a 2009 Museums and the Web paper, "Redesigning Your Museum's Web Site: A Survivors' Guide" (Burnette, 2009). By continuing the discussions we began in our first paper, which explored the shared joys and challenges we faced in planning, developing, and deploying new and improved institutional Web sites, we hope to offer the wider museum community an opportunity to tap into the first-hand knowledge and experience we gained from our projects. For the most complete picture of the full life-cycle of a redesign, our previous paper should be read in conjunction with this one.
The Organizational Impact of a Redesign
In the world of economics, there has been much discussion of the notion of "unexpected consequences" between a product or service, the consumer, and the supplier when bringing something to market. (Harford, 2007) This phenomenon is also evident in the case of museum Web sites, which don't exist in isolation. A museum Web site redesign can no longer be considered a tidy one-off project that exists on the sidelines, especially with the evolution from limited first-generation sites to today's dynamic and sophisticated platforms. Successful Web redesigns need to be viewed as systemic, institution-wide initiatives, as they touch upon almost every department within a museum and must respond to strategic goals.
Strategies: now you see them, now you don't
Web site redesigns are painfully effective at bringing larger and often unresolved strategic issues to the surface. For instance, at SFMOMA, the redesign project brought attention to the need for an institutional re-branding. In an ideal world, the institutional re-branding would have preceded the Web redesign, but the reality was that the SFMOMA Web team had to complete the Web site redesign by a specific fiscal year, and waiting for the re-branding would have meant delaying the project. As a consequence, they found themselves having to make brand- and identity-related decisions along the way without a formal re-branding initiative in place. The firm that did the Web visual design, Hot Studio, had to develop a new visual vocabulary without the structure of a larger re-branding project. MoMA began a larger institutional re-branding after the Web site redesign had begun, and consequently the Web team had to go in and make significant changes to the site design late in the project to accommodate this shift. NG, in contrast, had completed its re-branding efforts before the Web site redesign began; however, the focus had been on print application, so the redesign had to develop and adapt the brand further for on-line use.
In addition to the absence of a formal re-branding initiative, the SFMOMA redesign highlighted the inefficiencies of some of the institution's existing internal systems. During the redesign, the Web and collections management teams realized that the collections management system and the Web CMS were not going to play well together. The process of exporting data from the collections management system and preparing it for ingestion into the Web CMS is cumbersome and inefficient, and this process is still being fine-tuned even a full year after launch.
At NG, the launch of the new site also highlighted the somewhat "siloed" approach to overall organizational planning and inefficiencies in cross-departmental processes. Along with an increased emphasis in the museum on digital initiatives, there has also been a significant shift towards an appreciation of the Web site as a critical communication tool, and this has been reflected by the steady increase in demand for on-line publication. However, NG has not yet adequately addressed the long-term implications of maintaining an increasingly large site with limited internal resources. Decision-making processes and the approach to content development and publication have all been thrown into sharp focus as NG's leadership re-evaluates the organization's overall direction. Several new initiatives are now underway to define a way forward and establish a shared understanding for NG's digital future.
At both NG and MoMA, we had working digital strategies in place for our redesigns and were developing our Web sites against these frameworks. Since the launch of our sites in 2009, however, both of our museums have seen an increased focus on creating overarching institutional digital strategies as more areas within our organizations have become invested in digital initiatives. However, with this increased emphasis on digital initiatives, the challenge is to maintain a cohesive on-line presence rather than a disparate collection of projects. Creating a manageable structure that can grow organically takes planning and time - something difficult to explain to staff who are unfamiliar with Web development.
As the leaders of these types of projects, it's our job to keep an eye on the larger strategy as best we can (even if a formally sanctioned one does not exist), find do-able ways to tackle messy inefficiencies, and be the champions for change. Of course, it's equally important to recognize that a new Web site won't fix all organizational problems and that there are only so many strategic and process issues that you can take on as part of your redesign.
You build it, but what if they don’t come…?
Without high-level buy-in and larger strategic directives, organizational attitudes won't change with the mere fact of a new Web site. For example, at SFMOMA, the Web team made the faulty assumption that a CMS-driven Web site would inspire departments that are not normally involved with the Web to take more ownership. But a new Web site can't change personalities or institutional culture, nor can it make staff voluntarily take on more work. If there are departments that are not vested in your site, a new site will probably not change that, unless it is accompanied by redefined priorities coming from the top-level leadership.
At SFMOMA, this meant that some of the content areas of the site that had been designed to feature regularly changing content were scaled back because there was no larger institutional investment to develop the relevant content. In MoMA's case, while a blog was included in the initial structure and layout of the new site, it took several months post-launch for an editorial body and advisory group to form, take ownership of the project, and drive it forward to make it viable.
Organizational Expectations and Attitudes
Educating staff at all levels of the organization on what is involved in maintaining a new, expanded, and more complex Web site should not be underestimated. You’re most likely to find little or no interest in gaining an appreciation of the complexities and effort involved in creating digital content for a new site. It's easy for your colleagues to see your updated visual design, but much harder for them to "see" and appreciate the layers of expanded content, new information architecture, and behind-the-scenes code, metadata, and tagging that is consuming your Web team. This lack of understanding of the complexities of on-line publications will inevitably lead to moments of frustration when trying to balance increased expectations against what’s actually possible.
There is a common misperception that just by redesigning and adding a CMS alone, the Web team should miraculously be able to take on even more work. In fact, the notion that extraordinary levels of efficiency can be achieved without additional staff is a heavily overplayed card when seeking approval for a Web site redesign, but the truth is much more nuanced. While the Web teams at all three of our museums are now doing many things faster and more efficiently with the help of our sophisticated back-end systems, it’s debatable whether we are actually able to do "more" work. The nature and quality of our work has changed, our new sites are richer and deeper, and we are all working more strategically and creatively. More time is spent on tagging content and creating inter-relationships between content. For example, all three of us developed richer and far more complex calendar event pages as part of our redesigns. However, in doing so, our Web teams are now spending more time updating our calendars than we did on our old sites.
None of our museums added permanent staff to the internal Web teams after launch, and we’re still working with various freelancers and consultants involved in our redesigns as budgets permit. As our core teams have not increased, it has made life particularly challenging, as we find ourselves having to finish many of the bugs and features left over from the redesigns – ostensibly "invisible work" – while completing new projects at the same time. The reality is we are being expected to do significantly more work with essentially the same, if not fewer, resources.
There is a leadership and communication challenge here in managing organizational expectations about redesigns (Barker and Cole, 2009). In hindsight, all three of us felt that more effort could have been put into this exercise during the projects. Without educating key stakeholders in your institution about what it takes to get content on-line, and then developing clearly articulated institutional priorities, there is going to be continued frustration when Web projects, including the site redesign, can't be completed in record time. This situation is likely to be further exacerbated by the impact of the economy on budgets, as more and more institutions see the Web as the “cheap” or “free” alternative to print. As resources are finite, it’s imperative – in order to balance organizational priorities against audience needs – that digital teams be better supported in their efforts to work "smarter, not harder."
What a CMS Can and Can't Do
It's vitally important to be realistic about what your CMS can and can't do. We all discovered, although to varying degrees and with differing results, that our CMSs were not for the technologically faint of heart. As mentioned earlier, we also discovered that a new Web site with a sophisticated CMS back-end and complex data relationships demands more of the Web team and can often mean more time to prepare content for the Web, not less.
While a CMS may make it technically possible for departments to contribute content directly into the system, the reality is that CMSs are too complex, and the learning curve too steep, for the occasional user – especially when it’s not a key part of his or her daily work. Furthermore, the model of the Web team as "gate-keepers" does not go away simply because with a CMS it’s possible, in theory, to allow multiple authors to contribute content. On the contrary, distributed authorship necessitates even more editorial oversight to ensure consistent quality and cohesiveness of on-line content.
Several months after launch, the SFMOMA Web team was generally successful in its efforts to train select staff members on the CMS (SmileMaker by Carbon Five, http://www.carbonfive.com/view/page.basic/solution/content.solution/cms), with about a half dozen staff actively inputting content. However, SFMOMA learned that the more staff members who use the CMS, the more quality assurance work there is for the Web team in cleaning up invalid code inadvertently introduced by novice users. It took SFMOMA much longer to get staff trained and using the CMS than they had envisaged, and the reality is that they have fewer staff using the CMS than originally imagined. Staff members who have only occasional updates continue to use a simple Web form to request updates to the site, and the Web team makes the updates for them.
Fig 4: Exhibition page on the SFMOMA Web site, http://www.sfmoma.org/exhibitions/397
Fig 5: CMS screen for exhibition page on the SFMOMA Web site
At MoMA, only the Web team, which includes one editor, is using the custom-built CMS (developed in house using Ruby on Rails), but various staff members are using a WordPress back-end that connects with the site's front-end (also built in Ruby on Rails) to contribute content to the blog. This WordPress back-end was built so that staff could author in a familiar environment, while the content would feed into the same system used throughout the site for easier integration. This approach also allowed the Web team to observe how these administrative tools were used before expanding the custom tool suite for additional users. The approach MoMA has taken has been to build only those tools needed, and only when they are needed, in order to avoid an overly complicated, under-utilized system.
Fig 6: Exhibitions page on the MoMA Web site, http://www.moma.org/explore/exhibitions
Fig 7: CMS screen for exhibitions page on the MoMA Web site
In contrast, at present NG has chosen not to roll out the CMS (Amaxus, http://www.amaxus.com/, by BoxUK) beyond the Web team. Significant questions as to future demands on the team need to be addressed before any attempts are made to engage staff beyond the department. A further consideration is the technical expertise of staff beyond the team, and the degree of training that would be required for them to use the current system. One of the options being considered is to have staff use a simplified CMS interface for submitting material. Another potential approach would be one in which "super users" on the Web team have access to the full CMS interface; frequent contributors on staff have a simplified CMS view; and occasional contributors can use simple forms or continue with current "low-tech" workflow processes, such as e-mailing Word files directly to the Web editors. Whichever solution is adopted, it is evident that considerable effort is required to define new workflows and to adapt existing ones in order for staff to work most efficiently.
Fig 8: Event listing page (left) http://www.nationalgallery.org.uk/whats-on/next-4-weeks/ and exhibition page (right) http://nationalgallery.org.uk/whats-on/exhibitions/kienholz-the-hoerengracht
Fig 9: CMS screen for exhibition page on the NG Web site
For SFMOMA and NG, moving from the tradition of hand-building pages to using a fully template-driven CMS has involved a major learning curve for each Web team. Dealing with content in its most granular form (every link, every asset, every tag, every bit of metadata, etc.) has required significant changes in conceptual understanding of what content is, and has impacted working methodologies. Some tasks can now be achieved incredibly quickly, while others are time-consuming and extremely complex. In MoMA's case, they are using a combination of CMS-driven pages and static template pages that use widgets and other Rails mechanisms to share repeated content across different sections of the Web site. This means that certain custom pages can be developed easily without breaking down every single section into its individual components, while a framework still guides the whole site. In all cases, however, the teams at all three of our museums have had to become much more systematic and rigorous in the administration of our sites. Guidelines and documentation have been or are in the process of being developed, tracking and monitoring systems reviewed or established, and individual working practices codified.
Measuring Success: Statistics and Lies
A key component of any major project will be your ability to recognize whether you have met your goals, and to do that you'll need to be able to measure or quantify your "success" against agreed-upon targets. However, this is remarkably easy to say, and surprisingly difficult to do. For a Web site, these measures tend to encompass a mix of both qualitative (e.g. on-line surveys, focus groups, etc.) and quantitative (e.g. statistical usage) data. This data often forms the basis of ongoing benchmarking for governmental, sector, organizational, and funder reporting. But perhaps more importantly, it can also be an invaluable tool for decision-making among the various teams tasked with the ongoing maintenance and development of a site, as it can provide insight into the ways your audiences are actually using your site and consuming your content.
Valuable knowledge was gathered by all of us through a combination of direct user feedback and key usage trends. Statistical data, in particular, provided an indication of significant trends in terms of popularity of specific content, clarified key user journeys, and highlighted popular search terms within both Internet search engines and the internal search facility of the Web site.
That said, we would caution others embarking on a similar exercise to note some of the potential risks when trying to glean an understanding of usage if you are basing it solely on statistical data – especially when dealing with navigation or content. One particular risk occurs if there are underlying flaws in the way the data is collected (for example, log file analysis vs. cookies) or if this data is not "clean" (for example, not filtering out spiders or internal traffic sources). Without spending sufficient time ensuring that your data is as robust and reliable as possible, it's very easy to get a misleading impression of how audiences are interacting with your site. To minimize this risk, all of us adopted a "holistic" approach whereby we cross-referenced and tested our assumptions using a range of analysis techniques, including on-line surveys, stats-based data, and user testing.
While the focus of a redesign tends to be primarily on improving the front-end user experience, there is a significant opportunity to review and improve the behind-the-scenes administration of the site at the same time. A noteworthy part of this is to re-evaluate the role that ongoing monitoring of the site plays and the level of investment – in terms of both time and money – worth putting into this.
In the immediate pre-launch period, all of us had been regularly monitoring our sites using both Google Analytics and an additional analytics program. This was a reflection of a general frustration with the limitations of site usage data, combined with the challenges of finding a usable piece of software with which to analyze it. SFMOMA had been using WebTrends Analytics (http://www.webtrends.com/Products/Analytics.aspx), a hosted, fee-based solution, which, although extremely powerful, was also extremely complex to set up and manage. MoMA had been using Omniture SiteCatalyst (http://www.omniture.com/en/products/on-line_analytics/sitecatalyst), a similar hosted, fee-based system that is equally tricky to configure. NG, in contrast, had been using a log file analysis solution called LiveSTATS (http://softwaresolutions.fibre2fashion.com/company/deepmetrix/productDetail.aspx?refno=1154) for a number of years. This had been maintained by their external hosting company. In tandem with these various packages, all of us have been using the free Google Analytics service (http://www.google.com/analytics/).
In a post-launch environment where time and money are both in short supply, we all have decided to reduce the overhead of maintaining two data collection systems in parallel, and are shifting our efforts entirely to Google Analytics. This is because it's generally felt to be easier to implement, the data it collects appear to be more reliable, and it can provide comparable figures across organizations (this is especially true in the UK, where it is fast becoming the de facto analytics solution in museums).
There are, however, some inherent risks when moving from one analytics solution to another. We all experienced various issues when switching between systems. For the NG, the difference was in the way the systems collect data (log file vs. cookie based). For both MoMA and SFMOMA, the risk has been the loss of access to historical data when moving away from the fee-based solutions.
Most significantly, and irrespective of the analytics system you chose, the other key challenge around analytics inherent in a redesign is the break in continuity of data. With a new site, you are not comparing "apples to apples," as your site structure and content have invariably changed. That is why it's critical to spend time carefully considering the type of data that is going to be truly useful before you spend countless hours implementing tracking that you may ultimately not use (Kaushik, 2007).
Can a redesign "super-size" your traffic?
Increasing on-line visitors is often cited as a key argument for a Web site redesign but, in truth, it is somewhat disingenuous to expect a redesign in and of itself to be a driver for increasing traffic (Chan, 2008). Immediate post-launch figures for all three of our museums were a mixed bag, showing steady increases in some cases and slight declines in others. However, there was consistency in that all three of us were still seeing the familiar trend curves in the ensuing post-launch period (the highest traffic in the fall when school starts, spikes in traffic when there was a popular exhibition, etc.). Increasing Web traffic often seems to be a bit of a dark art, and relies on far more than the mere fact of a new Web site: on the quality and relevance of your content; successful search engine optimization; general trends in on-line usage and cultural consumption; and, last but not least, the popularity of your exhibitions and programming. If a key goal of your project is to increase traffic to your site, then sufficient time and resources should be allocated to improving the breadth and depth of your content in the pre- and post-launch periods.
Redesigns Never End
So, at some point in the redesign process, you've drawn a line in the sand and called it the launch date. This is a necessary stage in the project: it gives everyone a common goal and prevents the process from drifting aimlessly forever. But does a launch date mean that you are actually done? All of us can respond with a resounding "NO!"
As discussed in our previous paper, your redesign launch is just the beginning. Once your new site is up, there are still three types of work left to do:
- Fixing bugs that were never fixed or that emerged after launch
- Finishing work that was put on hold along the way because of higher priority tasks
- Rethinking work that was already done based on feedback from users
SFMOMA, for example, had a list of over 100 open bugs (tracked using JIRA, http://www.atlassian.com/software/jira/) and 30 unfinished features (in Pivotal Tracker, http://www.pivotaltracker.com/) at the time of launch. This original list of bugs is now down to 20 and the list of features is down to 15, but the lists of new post-launch bugs and features are still each over 50. This is because fixing bugs and developing new features often introduce new bugs and the need for new features. It's a moving target. The goal is not to reduce these numbers to zero, but rather to prioritize and re-prioritize effectively along the way so that your team can focus on what's most important.
MoMA found that in addition to bug fixes (tracked using Lighthouse, http://lighthouseapp.com/, and Mantis, http://www.mantisbt.org/), they were finishing components that weren't completed for the initial launch, as well as re-imagining and reworking sections that had already been developed for the redesign. For example, the site launched with a static, pared-down section on the museum's publications, but a database-driven publications section, with links directly into the store's shopping cart, was added a couple of months later. In addition, the MoMA Voices blog was renamed, redesigned, and re-launched several months after the initial launch, once an editorial group and content schedule had been established. MoMA also made some post-launch adjustments based on user testing and an on-line survey. Several of the changes made so far have been in terms of how the film schedule is presented, based on feedback from a very vocal, long-standing film audience. One of the overall ideas in MoMA's redesign was to present information in different ways to different types of audiences. Several months after this launched, while the basic idea still seems valid, MoMA realized through their testing that they need to do a better job of introducing or framing those different views. People seem to want a straightforward list as the default view, even if they have the other views as options. Work in this area is still in process at the time of writing.
Fig 10: MoMA Voices blog when MoMA.org redesign launched
Fig 11: Renamed and redesigned blog several months later
The NG Web team, six months after launch, is still allocating scarce time to addressing the remaining 45 bugs on the site (tracked using the vendor's maintenance system). With limited resources, this has significantly impacted plans to improve existing designs further and to develop new functionality on the Web site.
This post-launch phase can, in many ways, be the most challenging, as it demands a huge level of ongoing commitment from both the internal team and outside consultants – just at the point when everyone wants to be finished and move on to the next project! While a redesign is a rare opportunity to have everyone in a department working on a single project with a single goal, after the launch the tendency is to go back to the more fragmented workflow of different projects with different schedules. As a result, there can be a letdown from that loss of focus. After designing a new site, some of the other projects can feel pretty mundane, and it becomes necessary to find new and exciting projects for your team to work on.
Ensuring that you have sufficient resources to continue to address outstanding bugs is a major challenge, and can impact contractual obligations when key deliverables are tied to milestone payments. In the case of one of our museums, many of the bugs are still being addressed: this means that final payments haven’t been made to the vendor. This is particularly problematic as there is no guarantee that funds will be held over across fiscal years. Also, bug fixes usually have to take priority over improvements or new initiatives, but their "invisible" nature can give the outward appearance of little forward movement.
SFMOMA has been fortunate to be able to set up a retainer model with their developers at Carbon Five (the firm that did the site implementation and licenses the CMS), allowing for an ongoing relationship and continued support for both bug fixes and feature developments. SFMOMA still does weekly check-in calls with Carbon Five, thereby keeping in place the momentum and communication that was established during the redesign. This post-launch relationship has been incredibly positive and efficient due to Carbon Five's continuing commitment to working with SFMOMA, their close physical proximity, and the continuity of retaining the same staff who worked on the redesign.
Documentation is a key, and often overlooked, component of a redesign. All three of our projects used wikis for documentation (MoMA used Lighthouse, http://lighthouseapp.com/, NG used Microsoft SharePoint, http://www.microsoft.com/uk/everybodysbusiness/howmany/ collaborate.aspx?wt.srch=1, and SFMOMA used Confluence, http://www.atlassian.com /software/confluence/). While there is a usually a big push for specification and definition documentation in the first phase of a redesign, as well as decent efforts at integrating comments inside the code, there is significantly less attention paid to documentation after launch. And in many ways, this documentation may be more important, as it is a living, growing reference that informs all future development. The SFMOMA Web team has found it difficult to be disciplined in updating documentation post-launch, and getting the developers to maintain the documentation has been equally challenging. NG has also been pushing their vendors to add more documentation post-launch and has also resorted to scheduling in time for the Web team to be able to update their documentation on a regular basis.
Conclusion
In planning and launching complex, large-scale Web site redesigns, it is critical that you allocate resources and attention to the post-launch period. This often overlooked and under-considered phase of an on-line project can have an unexpected and significant impact on your organization's strategic planning, and can lead to dramatic changes in your Web team’s daily life. Complex new sites with content management systems present new and different challenges, and old working practices must be completely revisited. Managing expectations and measuring success are not easy tasks, and it’s not over when your site goes live.
Both SFMOMA and MoMA have adopted Agile software development methodologies, in which features are rolled out in small iterations on a continuous basis. Given how quickly technology changes, we have entered a new era in which we will always be in a redesign! And as new social media sites are integrated into our daily workflow, our sites need to expand and relate to those unforeseen initiatives that contribute to our comprehensive on-line presence. By moving to a more adaptable infrastructure, consolidating our information, and separating the front and the back ends, we’ve left behind the sweeping redesigns of the past and entered a new era of smaller, more rapid iterations of our sites. While still requiring a lot of thoughtful management and growth, this new era of museum Web sites is better equipped to adapt to the changing nature of museums on-line.
References
Barker, S. and R. Cole (2009). Brilliant Project Management (Revised Edition): what the best project managers know, do and say. Gosport: Prentice Hall.
Burnette, A., et al. “Redesigning Your Museum's Web site: A Survivors' Guide”. In J. Trant and D. Bearman (eds). Museums and the Web 2009: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2009. Consulted January 27, 2010. http://www.archimuse.com/mw2009/papers/burnette/burnette.html
Chan, S. “Towards New Metrics Of Success For On-line Museum Projects”. In J. Trant and D. Bearman (eds). Museums and the Web 2008: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2008. Consulted January 27, 2010. http://www.archimuse.com/mw2008/papers/chan-metrics/chan-metrics.html
Harford, T. (2007). The Undercover Economist. London: Abacus.
Kaushik, A. (2007). Web Analytics: An Hour a Day. Indianapolis: Wiley Publishing.
