TransformingTheChurch.org's Public Blog

Our Largest Upgrade -- Ever!

We’re in the final stages of the largest upgrade of our website that we have ever performed. Overall, we have worked on this upgrade for the last eight months.

There is nothing much left to do except tweaks on a few pages. We’ve gone all of our final quality-assurance work and now are only clicking through all of the pages on the website to make sure everything is correct.

The work began just before Thanksgiving 2014 when we migrated off our old streaming platform to the new system. The November upgrade maintained all of our legacy functions and added some capacity. However, the functionality implemented in this upgrade was pretty much just maintained at our legacy levels. To fully exploit the power of the new platform, we were required to do considerable software development.

Click here for details of the November Migration

We did deep preparation work for the full upgrade during all of the winter season to get ready. We began by building and testing our new and improved media platform. The media platform is a separate mostly-independent system so we could build it parallel to the production-grade website. The final test of this work was our very successful Online Film Festival that we conducted March 8-9^th. We used the Film Festival to generate a large load on the new system to stress test it. We served almost 7-thousand requests for the festival’s high-quality video files over a 57-hour period.

Click here to see the film festival.

We spent the spring season remastering and re-encoding all of our videos. With our upgraded media platform completed and tested, remastering and re-encoding of all of our videos were necessary because of our planned conversion to the HTTP Live Stream (HLS) protocol. As intended in our upgrade, we moved away from the RTMP protocol and the use of the Adobe Media Server (AMS) that we have utilized in the past. AMS and its RTMP protocol served us faithfully and well for the last decade (and before that in other projects), but HLS offered many advantages in our architecture.

Click here for platform details.

The switch required re-encoding each of our videos using our new cloud-based encoder service that is an integral part of our new media platform. To get the most quality out of the switch over to HLS, we remaster new intermediate masters from the original edit masters for every video on the website. Remastering was a very time-consuming process, and we didn’t finish with this phase of the work until late in June.

With new intermediate masters finished, we uploaded all masters to our new cloud-based storage system. Once uploaded, we used our new cloud-based encoding service to make eleven HLS distribution file sets for each video on the website, ranging in speeds from 328-kbps to 8,884-kbps.

• 8,884-kbps - spatial size 1,920 x 1,080 (29.97 fps)
• 6.884-kbps - spatial size 1,280 x 720 (29.97 fps)
• 5,384-kbps - spatial size 1,280 x 720 (29.97 fps)
• 3,756-kbps - spatial size 960 x 540 (29.97 fps)
• 2,256-kbps - spatial size 896 x 504 (29.97 fps)
• 1,756-kbps - spatial size 768 x 432 (29.97 fps)
• 1,392-kbps - spatial size 640 x 360 (29.97 fps)
• 1,192-kbps - spatial size 640 x 360 (29.97 fps)
•    792-kbps - spatial size 640 x 360 (29.97 fps)
•    528-kbps - spatial size 480 x 270 (15 fps)
•    328-kbps - spatial size 416 x 234 (12 fps)

With both the storage system for the masters and the encoding service in the cloud, reading the masters in the encoding process and writing back the finished files could take place a core network speeds (10 gigabits-per-second.) In turn, the cloud-based encoding service utilizes massively-parallel processing, which is ideal for tasks like encoding. The speed and quality of the encoding were impressive.

Our new cloud-based encoding service performed beautifully, and we accomplished all of the re-encoding in only four days of 24-hour-per-day work. Using our old local encoding farm, this would have taken more than six weeks to achieve.

The output of the encoding service automatically placed all encoded distribution files into our new Content Deliver Network to be ready for immediate use.

We then replaced our entire set of embedded media players with the newest version of the player and reconfigured to use the new HLS streams. We finished loading the new media platform and cutover to the new HLS streams for all videos in the early morning of July 10^th although we began phasing in with new videos beginning on July 1^st.

Starting late in the evening of July 10^th, we then began converting all of our pages to a new approach to supporting both mobile and desktop devices called “responsive design.” For mobile devices, we have supported another technical approach called “adaptive design” for some time. However, after extensive testing, the responsive design proved to be superior and have many operational advantages such as “authoring once, serving all.”

Click here for details about conversion.

We did the heavy-lifting of this conversion over the weekend of July 10^th. Come Sunday night, the public facet of the site was 98% converted to the new responsive design and working well enough to serve most of our visitors without major disruption. Any visible artifacts on web pages of the conversion were only minor formatting anomalies.

By July 18th, we completed the conversion of the public facet, so it was 100% correct.

For the last three weeks, we have converted all of the non-public spaces of the website such as our courses, lectures, communities and events.

For the last week, we have refreshed our search engine optimization and also re-proofed the entire site.

We’re almost finished, or at least as one ever is for this type of work. Tweaking pages tends to become a continuous process.

During the last half of August, we will be the adding new content that has been piling up while we have been working on the upgrade. To make the upgrade work go as quickly as possible, we pretty much made an all-hands-on-deck action. All of our web editors have single-threaded through the upgrade work to the exclusion of almost everything else. As a result, our normal posting of new content fell behind. During the next few weeks, we will be playing catch-up.

For example, just this week, we add in all of the "Chuck Knows Church" episodes, both for the original series and The Committee, that have come out while we worked on the upgrade. With the new media platform in full production, we were able to add the episodes with all of the new features our new platform allows, including Adaptive Streaming. BTW, if you don't know, we simply don't post the episodes, we get the original 1080p high-definition files and encode them, just as we do with our own videos, using Adaptive Streaming. Doing so allows us to stream Chuck with all of the quality that is possible.

Bringing Back This Blog

After several years of being inactive, I have decided to revive this blog in order to let our students, users and guest know what is happening behind the scenes of our website.

Give me a day or two begin writing some articles and I will be posting here.

Community Building Requires Interaction

You will often hear me discussing the definition of community that I use the most: "a community is a group of organisms interacting in a shared environment."

Since this isn't my own definition, I can brag on it; it is a beautifully elegant and simple definition of this complex thing we call "community." The definition perfectly fits anything from a coral reef, a pride of lions, a rural village, an urban city to our own Connected Community.

As it applies to our Connected Community, let's parse the definition in our context: "a group of organisms (that us; Disciples in the Southwest) interacting (participating in discussions in forums, sharing news, participating in collaboration and on-line learning) in a shared environment (in our case, in the on-line shared webspace we call the Connected Community.)

The Connected Community has been fully launched less than two weeks and had a soft-launch a couple of weeks before that. With the community platform functioning, we're now beginning a period of intense community building. For the next six months, we will be focusing a lot of attention on two particular aspects of our community building: (1) getting more "organism" into our environment; and (2) increasing the "interactions" within our environment.

Getting more organisms, more Disciples, into the community are actually the easy tasks of community building. Word will spread, people will join. Yes, you can help us spread the word and make the pace of this portion of our community building go faster, but it will happen.

The other portion of our community building will be the harder thing to do; that of increasing the interactions among the members of our community.

For this phase of our community building to be successful, we all are going to have to, well, interact. We have to share news and add threads and comments to our forums.

Although this is indeed as easy as it sounds; it is often difficult to get people started. There is a natural tendency in all of us to want to "lurk" in a new environment. Lurking is so common that we on-line community managers have a standard definition for it. A "lurker" is someone in a community who is a constant visitor/reader to forums but rarely if ever posts in them. By extension, a lurker is also someone who reads the news of others but never posts news themselves.

Lurking in a new space for a short time can actually be a good thing; it allows you to observe the community for a while in order to get into the swing of things before jumping into full interactions. However, at some point, you DO need to jump in and begin contributing in order to build the community through your direct interactions with others in the community. It's that "organisms-interacting" thing mentioned in the definition. Otherwise, we're not a community.

Participation in the community directly helps build the community while perpetual lurkers only benefit themselves, they don't help build the community only consume without contributing. While I don't want that to sound unduely harsh, I must acknowledge it is a simple fact. Please be assured that we're never going to castigate lurkers here; because lurking is a very common human trait particularly in a new community. Instead, what we want to achieve is to encourage people to change that type of behavior; to participate in community because they want to and because they see its benefits!

We're going to encourage you to participate in our community at every opportunity. At times, we even going to implore you to because the more people who participate, a.k.a. "interact," the more vibrant our community will become.

The normal cycle of interaction isn't a lot of work on your part. In a healthy community, some times people participate and some times they only lurk for a while. This is totally normal behavior. Everyone participates at some time but not necessarily all of the time. A healthy community has a balance between contributors and consumers. The base of contributors simly needs to be as broad as possible.

As we get the Connected Community started, it is very important that we have as many interactions within the community as possible early in this critical portion of its life cycle. These early interactions are the seeds that generate community building and more interactions down the road.

Participation is easy, simply pick a forum you're interested in and start adding comments to topics. Your comments don't have to be long; they simply needs to be some sort of interaction with other Disciples. You could even post questions in the topics. Better still, you could even start a new thread in one of the forum on a new topic you're interested in. Before you know it, you will be in discussion and that is indeed participation and interaction.

You'll be a community builder.

The more we interact, the more be become a community.