Let's rokk! [Tudor Cret's blog]

January 25, 2010

Behind ciripescu.ro, a Windows Azure cloud application (part 2)

Filed under: Technologies — Tudor Cret @ 2:14 pm
Tags: , , , , , , ,

 

Let’s go deeper(1) – web role and storage

As the high level design diagram suggests, the communication between web roles and worker roles is done through the various features of the Storage account, such as Queues. Queues are normally used to send messages/tasks between roles, as they implement all the safety features needed for asynchronous communication between multiple applications, running of different servers. This includes locking and recovery in case one of the roles crashes while still processing a message. Roles are automatically restarted when they crash, and items popped from the queue are made visible again. In fact this is the beauty of Windows Azure, from the perspective of the software developer: you write the code as if you only run one role and the cloud will make it work automatically with any number of roles, to infinitely scale it up. The beast is that Azure Tables are somehow limited, because there are no sorting or grouping options. But of course, there is SQL Azure (the classical SQL relational database – cloud version). Yes but sometime even at a relation database you make some denormalization for speed purposes. So it would have been nice to have at least sort option for Azure Tables.

While message passing between roles is mainly done using Azure Queues, other information are still passed by the traditional way: shared access to resources such as a database. Azure Store offers two different types of non-relational cloud based storage: Blobs (that store entire files) and Azure Tables (that store entities).

Ciripescu.ro uses Azure Tables to store all its business entities, instead of a classic SQL Database. We are still talking of business objects, a business layer and a data access layer, as you would in any other application. The only difference is that the data access layer stores the objects in a non-relational storage, designed to be using in a cloud environment. What makes Azure Tables so special is that it allows you to have very large tables that are still searchable very fast. In Ciripescu’s case, the table containing messages sent between users (Cirips) could have millions of items. Querying a sql table with a few million items can take several seconds. If we think of Twitter we would realize that a few million is more like a joke. How do you search among a table with a billion entities? The answer is simple: split that table on a cloud of servers, by using a relevant partition key and either only search that one partition where the object is, or search all them, but in parallel. This is exactly what Azure Tables does: for each object you define a partition key that will allow Windows Azure to transparently split the table, and a row key that allows fast searching of data in a single partition.

User entity

A business object would have to define those two fields and map relevant properties to them. Take note of the User class from ciripescu.ro, that inherits from TableStorageEntity (that defines the said properties) and maps the username as PatitionKey and String.Empty as row key, through its constructor. The TableStorageEntity class is defines in the Windows Azure SDK.  

In order to query an entity from Azure Tables, one must first create a DataServiceContext. This is a class that must inherit TableStorageDataServiceContext, which in turn inherits from the LINQ class DataServiceContext. Here, the conceptual object model looks like the one from NHibernate, but it’s not the same thing.  

User entity objects retrieval

Everything else is simple LINQ: the programmer creates a DataServiceQuery object for each Table and makes LINQ queries on it:  

var users = from u in Users select u;  

foreach (User u in users) { ….}   

This is how a data access layer class would query items from the storage:  

Quering items from the storage

Inserting, updating and deleting objects is done in the same way LINQ programmers are used  to. Let’s not forget that the data access clases from Azure SDK are build on LINQ classes:  

CRUD operations on User entity

The other entities in the application are built using the same object model.  

Now let us go to the main advantage of a social networking platform made on cloud based technology: the table that holds all the messages (Cirip). Social networking websites can get very popular. Think of tweeter: millions of tweets per day, and they have to not only be stored somewhere but also queried. How do you query a table with 1 billion entries? Azure Tables is the answer. Lets take a look at our Cirip table. Let’s assume two users, Tudor and Pitagora, had the following message exchange:  

Tudor: This is the first post on ciripescu   

Pitagora: hello world  

Pitagora: Windows Azure rocks  

Pitagora: What a good day for a Microsoft presentation  

Tudor: A barbeque would go better 🙂  

Tudor: I want a vacation  

PartitionKey RowKey Content
pitagora 9 What a good day for a Microsoft presentation
pitagora 12 Windows Azure rocks!
pitagora 20 Hello world
tudor 7 I want a vacation
tudor 8 A barbeque would go better 🙂
tudor 25 This is the first post on ciripescu

The PartitionKey and RowKey are two mandatory fields for any entity stores in Azure Tables. The programmer has to chose what he wants to store there carefully because all the power of the cloud depends on these two fields. Windows Azure splits tables in different partitions and stores them on different nodes, using the PartitionKey field. For redundancy and speed, each partition has 3 copies in the cloud. All partitions and copies of them are managed transparently by the cloud. The user doesn’t have to know of them are there. His only control over this process is the choice of PartitionKey. Entities with the same PartitionKey will belong to the same partition and be stored on the same node. The fastest possible query is the one that only searches in a single partition, so this choice depends from application to application. One has to ask: what is the most frequent query my application will do? In the case of ciripescu.ro, which is a microblogging application, our answer layed of the blog profile of each user. We decided that displaying all messages of a single user has to be the fastest query, so we chose the sender’s username as PartitionKey. In the example above messages from pitagora, can be stored on a different node then those from tudor. The order in which they are listed in the table also suggests that. Windows Azure sorted those messages by PartitionKey and RowKey.  

The second field, RowKey, is similar to the primary key in a SQL Table. Entities are index by it, and the fastest query done in a partition is one where the PartitionKey is the search criteria. In the context of Windows Azure, the Rowkey has another very important function: sorting. You have to remember that Azure Tables is not a relation database. It’s just of storage of entities. That means that you can’t run complex queries that involve counting, grouping and sorting. The entities are sorted ascendant by RowKey, and there is no way to change that. So in the case of the Cirip table, we want the latest Cirip to be displayed first. We chose DateTime.Max – DateTime.Now as a RowKey. The reason why I didn’t use real values in the example is because these are 11 digit numbers. The only thing I kept in the example is their order: the lastest message will always have the smallest RowKey.  

Using a non-relational database has some disadvantages too. Besides the lack of sorting and grouping capabilities sometimes you feel that you need relational entities. Because you will write more code at DAL layer. You can’t build :  

Select c from Cirip c, User u, Urmarire urm Where c.PartitionKey = u.PartitionKey and  

u.RowKey = f.RowKey and urm.PartitionKey = ‘current user’  

Anyway why Azure Tables and not SQL Azure? Because in our case is faster when searching. Also at the beginnings of Windows Azure there was not an SQL Azure. There was some kind of storage but not-relational. After some months and based on the feedback from developers Microsoft announced SQL Azure(summer 2009) which officially was released in the fall of 2009.

Worker roles and queues exposed here.

Behind ciripescu.ro, a Windows Azure cloud application (part 3)

Filed under: Technologies — Tudor Cret @ 2:13 pm
Tags: , , , , , , ,

Let’s go deeper(2) – worker role and queues

As mentioned above a worker role can be compared with a windows service. A little thing that sits quiet and it waits for its job. You tell it what to do, it knows how, you give it some input or not and it does its job. Also a worker can be seen as a background unit ready to execute a task. It depends on each one imagination and needs on how worker roles are used. Currently ciripescu.ro uses a worker role for email sending. It represents a “smart” email sender. Why smart? First of all we import a list of yahoo contacts to which an invitation has to be send. A person has 200 or 300 contact emails in his/her list. You can’t block the user until you send those emails. Also if you have online 10 users then you’ll have to send 2000-3000 emails in the same time. But what are doing if you have 500 users in the same time?  Also another problem appears. Yahoo for example banns you, if you send a large amount of emails per minute from the same email addres, even if you sign that emails. It sees that emails as spam. To avoid all these in the worker role we’ve built a scheduler for emails sending, of course giving priority to urgent mails such as ‘Welcome email’  or ‘Password reset. We’ve used queues to “tell” the worker the recepient(s) and the body of that email. More about Queues and communication can be found in Windows Azure training kit and Windows Azure SDK.

Azure Queues are a wonderful mechanism. They allow the communication between process threads or roles and they are based on a producer-consumer system. I can say that somehow they resolve the concurency problems of a distributed system. Ciripescu.ro uses one queue for sending urgent emails and another one for the rest of the emails, not urgent – that are sheduled upon a timetable. In the figure below you can see defined queues and how a new item (invitation message) is added to the queue, in web role:

Adding a new item to Azure Queue

Now the worker will consume the item recently added and it will deliver an invitation mail to ciripescu.ro upon a well known delivery timetable, like in the figure below:

Consuming a queue item

 

First we get the queue used for sending invitations emails (service.GetQueue(…)), we check if the queue exists/has items(queue.DoesQueueExist()) and then we consume the item by sending and email and deleting the item from queue(queue.DeleteMessage(..)).

In the beginnings of Windows Azure(winter 2009) a worker once started it would entered in  sleep state if it has nothing to process, or we needed to have an up and running worker all the time in order to process emails. For this an infinite loop was created (‘while(true)’), and the loop was triggered when the worker starts. In the last release of Azure this is resolved and all code inside the infinite loop should be passed to Run()  method defined by the role.

Azure costs…a lot of money from a developer point of view. And because at the beginnings of Azure the number of workers was limited to one, we’ve found a trick to limit the numbers of workers –  now is good to save some money.  What if when a worker starts you start more threads, each thread processing what a worker would normally do? Of course that you lose in performance, but it is a solution….

Outro or instead of conclusions

 Microsoft announced that upgrade from Community Technology Preview (CTP) accounts of the Windows® Azure™ platform (i.e., Windows Azure, SQL Azure and/or Windows Azure platform AppFabric) to paid commercial subscriptions is available, starting with January 4th. And IT COSTS.  The problems about the costs are relative. How much can an Azure application costs you can be calculated here using TCO and ROI.  Probably it’s a long discussion about which companies are going to pay for Azure, who is going to develop applications using Azure (since testing in the real cloud costs…etc) and more pages are required for this analysis. But for sure Azure is dedicated to enterprise applications,  to large companies which don’t want to invest more money in their own hardware and datacenters, which doesn’t want to invest more in sys admins. It’s sure that those may watch Azure as a storm in their activity. It’s sure that in the next years the biggest applications of the world will be in the cloud. Not only Micro$osft cloud…the others too. It’s sure that for small companies it will be hard to develop Azure applications.  We want to see Microsoft that comes and says: “No….it’s not sure…because…at least for the pricing…”. On the other hand Microsoft invested a lot in data centers and they must recover their money. Also you don’t need the cloud to host a simple web application with a small database in backend. There are enough companies to do this almost for nothing.

 Also it’s sure that www.ciripescu.ro will live as long as Azure will be in CTP. It’s sure that we liked to work with azure and to develop this application. I say “WE” because we are two of us who plays with Azure for almost one year. And here I want to thanks to my  colleague and friend Tudor Carean for his work at this project. 

In the end I would point what we liked most at windows azure, but also what we don’t like so much, when building http://www.ciripescu.ro:

  • Azure costs….a lot. For Ciripescu.ro it costs too much to pay after the CTP account will be disabled. It’s better to develop a non-cloud version and to host it on-premises.
  • We don’t like Azure Tables storage model(used at ciripescu.ro), because sometimes it is too rigid.  It is a good model, but it could have been better.
  • When deploying the application to the cloud it takes some time. The whole deployment mechanism is a black box process. You are not sure how much time to estimate with deployment.
  • Azure is  a new paradigm (at least for us),  it introduces the concept of cloud and cloud computing.
  • We like the idea of “worker role” and “web role” and the possibility of building hybrid applications .
  • Azure adoption was easy, it happened for just one time to reinstall windows because of Azure CTP kits 🙂
  • We like the idea of scalability and how it is implemented – “Azure is one click scalable”.
  • Availability and durability are also strong points of Azure.

Probably there are more others too…

Source code of the entire application is available on codeplex at http://ciripescu.codeplex.com/ .  For additional informations please contact me using contact details from the about page or send an email at contact [at] bitstarsolutions.com.

November 13, 2009

Developing ASP.NET web applications with SharePoint 2007 and CSLA.NET(part2)

Filed under: Technologies — Tudor Cret @ 3:50 pm
Tags: , , , , ,

In the first part we have seen which are the requirements for developing an ASP.NET application for SharePoint 2007 and we build a demo application using CSLA.NET. We’ll continue by customizing and deploying the application we’ve created.

Customize ASP.NET application for SharePoint 2007

Before we start it is good to know what is our purpose, what we have until now and what we are going to do :

  • We want to deploy our ASP.NET application created earlier using CSLA.NET on SharePoint 2007
  • We’ll have to use Visual Studio extenstions for Windows SharePoint Services in order to create an Empty project that will help with automatic packaging and deployment to SharePoint Server
  • Link the pages from the Web Application we already have with the newly created SharePoint project
  • Do some configurations on the projects and config files to work with SharePoint

 A good resource about developing custom ASP.NET pages with SharePoint 2007 is Jamil Haddadin’s blog.

We’ll start by creating a new Empty project that I named CSLADemo:

 a6

Figure 5 Creating VSeWSS project

 In this moment the solution contains applications projects and the empty VSeWSS project:

 a6

Figure 6 Solution explorer before customization

The next step requires adding a new web form to the web application MyShop2. I called this web form MyDemo.aspx. It will implement demo functionalities plus some special configurations specific to SharePoint.

 a7

Figure 7 Adding a new web form

 Having created our demo web form a new Module  project item must be added to the CSLADemo project. Also the recent created MyDemo.aspx web will be added to this module. In order to avoid any duplications we’ll add this using Add as link feature. This module is specific to SharePoint and it represents the element through which our custom web forms will be deployed to the server. Be sure that you modify the namespace in MyDemo.aspx.cs, MyDemo.aspx.Designer.cs and MyDemo.aspx to CSLADemo.

 a8

Figure 8 Creating a new module

 a9

Figure 9 Add MyDemo.aspx as link

Now our solution looks like:

 a10

Figure 10 Solution state

 As you it can be observed I  used a key to sign dlls we’ve created earlier: libkey.snk, dallibkey.snk. If not, SharePoint will throw a runtime error. Csla.dll is already signed.  Also these dlls must be added at CSLADemo project, as link, if not they will not be copied to the deployment directory targets.

In  Module.xml a File element has to be included. Update the Path and Url attributes to the file name of the .aspx file. This deploys the page when the SharePoint solution is deployed, in my simple case; the Module.xml will look like:

Path attribute refers to the reference path of the file in visual studio project and URL refers to the URL of the page in the SharePoint site, in this case our page will be deployed here http://ServerName:port/MyDemo.aspx

  a11

Figure 11 Module.xml editing

 In the Configuration Manager of the solution, clear the Build for theCSLADemo check box, see the picture below..

 a12

Figure 12 Build configuration

Add Safe control  entry for each deployed dll in manifest.xml. The file can be found in solution’s WSPview .

 a13

Figure 13 Adding safe controls

 To deploy the solution change the Start browser with URL in the CSLADemo properties page in the debug pane with your SharePoint root site, in my case http://vpc2003/

 a14

Figure 14 Project starting settings

 Before deploying the application and testing it we must comment the SqlDataSource tagMapping in web.config  of the SharePoint application, the web.config associated to http://vpc2003/. It can be found in the virtual directory root.

 a15

Figure 15 Comment data source tag mapping

More about customizing asp.net pages can be found on Jamil Haddadin’ blog.

 Hoping that no errors occur, deploy the solution and test it calling http://vpc2003/MyDemo.aspx

 a16

Figure 16 Running application(1)

 a17

Figure 17 Running application(2)

 The solution is available for download here: http://bitstarsolutions.com/FirstCSLA.rar

Resources:

Developing ASP.NET web applications with SharePoint 2007 and CSLA.NET(part1)

Filed under: Technologies — Tudor Cret @ 3:49 pm
Tags: , , , , ,

I will try to summarize my experience with SharePoint development. We’ll see how we can develop a custom ASP.NET web application that is built using CSLA.NET. CSLA.NET is an application development framework  intended to help in reducing the cost of building and maintaining applications, but it depends where  and when it is used. With CSLA.NET we easily can use the same business tier for different presentation tiers (windows, web, web services ).

 SharePoint Development

 SharePoint is nothing but a smart ASP.NET 2.0 application which extends the ASP.NET framework by virtue of installing couple of custom components such as HTTP module, HTTP handler, Virtual Path provider, etc.

The end result? We have a provisioning engine that could be used for creating web sites based on the templates that come with it. Of course SharePoint offers more features but those are for some other day.

 So what do we exactly mean by “SharePoint Development”?  That means we want to write some code to extend the out-of-the-box functionality that comes with SharePoint. We’ll focus on adding  custom ASPX pages that help users accomplish a business function. In other words we’ll focus on integrating our custom business pages into SharePoint. Behind those pages we have business rules and processes. For example, a set of Order Entry pages which walk the user through a series of custom ASPX pages for gathering some data. In the example I choose to implement, I simulated the minimal functionality of a shopping cart, concentrating more on the code behind and the business processes.

 CSLA Development

 CSLA stands for Component-based, Scalable Logical Architecture and it is a software development framework that can help in building a maintainable business logic layer for Windows, Web, service-oriented and workflow applications. CSLA.NET enables you to create an object-oriented business layer that abstracts and encapsulates your business logic and data. The business objects supports binding for all major technologies like WPF, Web Forms, Windows Forms, Silverlight etc. Also by using CSLA.NET gives you the possibility to use easily the low coupling principle for the business, presentation and data access tiers. CSLA.NET is perfect to use when you need the same business tier for multiple presentation tiers, or when your code runs against multiple machines, you work in a distributed environment. But all these benefits are  time costing. The adoption of CSLA.NET is not easily made and requires some time. Becoming efficient with CSLA requires more time, time in which you can use alternatives which depends on the business process and environment you are going to work with. More about CSLA.NET can be found on the official site, here. The best resource to learn CSLA development is Rockford Lhotka’s (creator of CSLA.NET) book Expert C# 2008 Business Objects, also available in VB.NET version. Demo application presented in these book is available in CSLA.NET distribution.

Tools, technologies and environment preparations:

Before start proceeding with development be sure that SharePoint is up and running on your server and also Windows SharePoint Service works fine. You can do this by running your Central Administration Panel from SharePoint start menu item and access your installed sites.

Also be sure that you installed CSLA.NET Visual Studio templates. They come with the CSLA distribution and it helps in creating CSLA projects and entities.

  Developing ASP.NET applications using CSLA.NET

 In the next section I will focus on developing an ASP.Net application that uses CSLA.NET and that will be deployed on SharePoint. The application simulates the functionality of a shopping cart (adding products, ordering and submitting orders) for an online store. We’ll start up by building our database:

  a1

Figure 1 Tables diagram

 Product table contains data about the available products, size, price, name and description. A product can be ordered by an user and it will be associated to an order, represented by Order table. Products are associated to an order using the OrderDetail table. The shopping cart is represented by an Order with status “N”. In Order table we can not have more than one order having the status set to “N”, since we have only one shopping cart. When the products from shopping cart are submitted we’ll just change the status of the order to “P”.

I did not included here any reference to users and user roles because I just wanted to simplify the business model, in order to understand better the new paradigms provided by CSLA.NET, but the model can be easily extended at any time.

Having the database already done, we’ll continue by creating a new Web Application within a new solution using Visual Studio 2008. Let’s call the solution FirstCSLA  and the web application MyShop2. Also we’ll add two more class library MyShop.Lib  and MyShop.DalLinq which represents the business layer, respectively the data access layer. We’ll try to obtain a low coupling relationship between those layers. In the figure below you can see the solution and the projects we’ve created until now:

 a2

Figure 2 Created solution and projects

Data Access Layer(DAL)

At this time I’ve chosen to implement DAL  using Linq, by simply adding a new Linq To SQL Classes  item to MyShop.DalLinq project as in the figure below:

a3

Figure 3 DAL Linq implementation

 To generate MyShop.dbml  create a new database connection from Server Explorer and  just  drag and drop tables and stored procedures into the designer window.

Business Layer(BL)

This layer contains one of the more interesting parts and it uses the new CSLA.NET elements and paradigms. Until now we have done DAL and database and we know that we need a low coupling relationship between  BL, DAL and Presentation Layer. This problem will be resolved using CSLA.NET. This framework is created in a such way that will help developers and architects to split the architecture of the entire application  to accomplish their needs and requirements with minimum costs.  As an example let’s suppose that the BL  it’s so complex that we need to run it on multiple machines, in a distributed environment. But in the same time we can keep our DAL and Presentation Layer on the same or different machines. More about the applicability of CSLA.NET can be found in the first chapters of  Expert C# 2008 Business Objects book.

So, I started to create the classes that I needed in dealing with ordering process and shopping cart. First of all we need to retrieve the list of products. Without CSLA.NET probably we would have been using the MVC pattern and we would have been created a controller to retrieve this list. Or more easily using a SQLDataSource or LinQDataSource in the Presentation Layer. With CSLA we’ll create a read only list that will contain the business objects, in our case the products that exists in database. I choose to implement a read only list since I had no intention to implement the edit feature for the products. Also we can use a simple read only business object in this case. So, I’ve created ProductLIst  class that handles ProductInfo business objects, as you can see below.

a21

GetProductList() method returns all the existing products in database and watching the fetch method we can see how DAL is used to retrieve the products. Also we observe that  in class definition

“public class ProductList : ReadOnlyListBase<ProductList, ProductInfo>”

  we have specified the type of the business objects – ProductInfo, which contains the properties associated for a product.

a22

I continued by creating the Order class and the list that handles the order business ojects. This time we have to edit an Order because we have to implement the submission process that suppose an order update. For this reasons it is necessary to register  its properties, in order to let CSLA to maintain the object’s state:

a23

More details about property registration can be found in Expert C# 2008 Business Objects book.

In the end I implemented the classes necessary to hold the relation between an order and its products. I named the business class OrderProduct and the list class OrderProducts,  which in fact logically maps on the many to many relation between Product and Order tables in database.

The complete class diagram for the BL is shown below:

 a24

Figure 4 BL classes

 

Presentation Layer(PL)

I will concentrate on data bindings and how we use classes from BL to display and operate with data. First of all it’s important to mention that CSLA.NET provides a built-in data source object, CSLADataSource that extends the functionality of .NET built-in ObjectDataSource. More about CSLADataSource object can be found in the Expert C# 2008 Business Objects book.

But why CSLA provides this built-in data source object ? As you could have seen I have implemented some lists in BL that handles different business objects. Now using CSLADataSource is very easy to bind these lists to controls : DataGrids, DataViews, DataLists. I choose to use a DataGrid. For example the binding of the products to a grid view looks like:

 a4

Binding a list of business objects to a grid is as simple as making two drag and drops and setting up the type name and the assembly of the list for the data source. If we watch closer to the datasource definition we observe that  TypeName=”MyShop.Lib.ProductList, MyShop.Lib”  “tells” what type of objects will have to handle with. OnSelectObject  event binds the list with the business objects to the data source. Behind the design, the only thing we must assure is that during the  page load we call grid’s databind method and that we select the business objects datasource.

a5

 I used the CslaDataSource for the other bindings too. The rest of the code, with the entire solution is available for download here.

In the second part we’ll see how to customize and how to deploy the application we’ve just created.

March 11, 2009

Windows Azure

Filed under: Technologies — Tudor Cret @ 1:37 pm

In the autumn of 2008, after TechEd Barcelona all admins were a little bit disappointed, I can say they thought about their future, what are they going to do after Azure will be official released.  And of course the answer is that they will continue their work, because we are going to have on-premises application for a long time since now.

Three words that describe Azure are “scalability, availability,durability”. Solutions on Azure are scalable, Azure services are always available – we have no database, no try/catch on SQL queries in our C#/VB code and of course everything is durable on the cloud. Windows Azure is a cloud services operating system that serves as the development, service hosting, and service management environment for the Azure Services Platform. Windows Azure provides developers with on-demand compute and storage to host and manage web applications on the internet through Microsoft data centers. There are 5-6 green data centers all over the world. Right now a data center from Chicago supply warming for the houses in the city.

Azure Services platform
Azure Services platform

 As we already pointed Windows Azure is the operating system that serves as  the development, service hosting and service management environment for the Azure Services Platform. You can find more about them here. Azure supports other platforms too, like Java, PHP, Python.

Live Servicesare used for handling user data and application resources. For example using Live Services you can build rich social networks, across multiple devices that can connet throungh internet. In my opinion Microsoft SQL Services takes a very important role since they are a Web-based distributed relational database into the cloud.  It provides Web services that enable relational queries and search. It can store and retrieve structured, semi-structured, and unstructured data. Et voila, nous n’avais pas une designe pour notre base de donne. Because of having SQL Services. 

Microsoft .NET Services make developing loosely coupled cloud-based applications easier. .NET Services includes access control to help secure your applications, a service bus for communicating across applications and services, and hosted workflow execution. These hosted services allow you to easily create federated applications that span from on-premises environments to the cloud.

SharePoint Services and CRM Services are used for collaboration and building stronger customer relationships and they will be developed in the future. More about how all these works can be found on Microsoft’s site.

Writing applications on Azure

Before starting to code your Hello World application on Azure be sure that you have a backup of your computer. You must have installed Windows XP SP3 or Windows Vista SP1. Then download Azure Services Training Kit, that contains resources that helps you in writing your first application and also a set of existing demos and presentations.

Download and install Windows Azure SDK.
Download and install Azure Tools for Visual Studio.
Download and install Microsoft .NET Services SDK.
Download and install SQL Data Services SDK (SDS).
For Java developers Microsoft provide Java SDK for .NET Services available here.

After installing and configuring your environment you can build your Hello World application or you can find it in Training Kit. Steve Marx presented at PDC 2008 how to deploy our first application on windows azure. The video can be found here.

Resources:

Official Windows Azure site:
http://www.microsoft.com/azure/videos.mspx

Videos and white papers:
http://www.microsoft.com/azure/videos.mspx
http://www.microsoft.com/azure/whitepaper.mspx

 Developer SDKs:
http://www.microsoft.com/azure/sdk.mspx

 Azure SDK:
http://msdn.microsoft.com/en-us/library/dd179442.aspx

October 24, 2008

Time for life coding…

Filed under: Others — Tudor Cret @ 10:58 pm

What happens when you realize that you have to go to the dentist ? First of all you are scared a little bit, or you have a panic attack or you simply say…NO…or you do not even want to think about it. But it comes a day when it happens to say YES. (And is not that day when you get married…). And you go the dentist, scared because of the chair and its tiny machines, the peoples in white near you…or just because of “phobic problems” . But what may happens so badly ? Watch this:

This happens, if you are Bean.

It was no surprise for me when it happened to have to go to dentist. Beware of Mr.Bean’s experience I have just imagined that I sit in F1 car, of course not an usual one…a red one… I was scared a little bit, I have to admit. But I met a wonderful and cool doctor – Dr.Razvan. He “killers” you with jokes, fun and he fills you with positive energy. Before all these he gave you a dental anesthesia in order not to be able to feel something when you laugh. It’s so cool… And in the end you will see that all your medical problems are fixed in the best professional manner, all your dental bugs are fixed, and you are  filled with a positive energy for the rest of the day. You enter at Dr.Razvan’s clinics worried about your “medical bugs” and you get out not worring anymore, just filled with positive energy and having a “bestial” job done by this cool doctor. I recommend him, it’s a great professional and a wonderful person.

Open GL

Filed under: Others — Tudor Cret @ 10:25 pm

Some days ago when I took a coffee break after a hard work at a site that will launch soon, I had the pleasure to be asked for a good tutorial for OpenGL. Of course that i did not resist and I start googling for a good tutrial. Ans look what I have found:
Have you noticed how most OpenGL Tutorial sites only talk about drawing and playing with triangles, quads and simple 3d objects (certainly not anything more complicated then cube) they draw using hard coded coordinates? That’s not exactly they way to go to create an impressive 3d scene that actually looks like a cool 3D game. If spinning a colored triangle is not good enough for you then you’ve come to the right place.”  And it’s right. All other tutorials talk about drawing and playing with triangles etc using hard coded coordinates. I really liked 3d coding tutorial and i recommend it, it is a first hand 3d tutorial.

 

http://www.3dcodingtutorial.com/

October 6, 2008

CCS 2003 & MS-MPI (part2)

Filed under: Parallel programming — Tudor Cret @ 11:27 pm

Now that we saw how to install and configure a Windows cluster (see part1 for instaling and configuring CCS) let’s find out more about MS-MPI and see how to run a parallel application.

Installing MS-MPI

MS-MPI is the implementation of the MPI standard, helping the users to create high performance parallel applications for Microsoft Windows Compute Cluster Server 2003. Microsoft provides a SDK, available here, that is used to create parallel applications. The Microsoft Compute Cluster Pack SDK contains executable binaries for Microsoft MPI and also headers and libraries for developing parallel applications. Also it includes API for integration with Microsoft Job Scheduler.

Download the SDK from the link provided above, 32-bit or 64-bit version and install it using the installation wizard either on a cluster or a simple machine on which a cluster behavior is simulated.

More about using CCP SDK can be found on MSDN – Using CCP.

More about MS-MPI can be found here. Also there are some MPI implementations for Microsoft .NET environment: MPI.NET and Pure MPI.NET.

Installing and enabling MPI Cluster Debugger

Visual Studio 2008 Professional Edition and Visual Studio 2008 Team System support remote debugging of applications, including parallel applications. The Visual Studio remote debugging process for a Message Passing Interface (MPI) application uses the following:

  • Msvsmon -the remote debugging monitor application of Visual Studio.
  • Smpd -the MPI daemon process. Starts mpishim.exe.
  • Mpishim -the application that connects to msvsmon.exe and that starts mpiexec.
  • Mpiexec -the MPI job launcher that starts the user’s application.

To use remote MPI debugging on a CCS cluster, you will need to perform the following tasks: 

  • MPI must be installed and configured on each node of the cluster.
  • The MPIShim.exe file must be installed on each node in the cluster, and in the same location on each node.
  • The Visual Studio Remote Debugging Monitor (msvsmon.exe) must be installed on each node in the cluster.
  • The Visual Studio host computer (the one from which you are debugging) must be set up with an account that has sufficient privileges to execute jobs on the cluster, and must be on a network segment and subnet that gives it access to the compute nodes of the cluster.

The Remote Debugging Monitor is specific to each processor architecture. It’s important that you install the x64 version. To install all the required remote debugging components, do the following at each compute node:

  • Insert the last disk of the Visual Studio 2005 installation set.
  • Navigate to the Remote Debugger\x64 folder using Windows Explorer.
  • Double-click rdbgsetup.exe to install the remote debugging components.

Debugging cluster applications

  1.Prerequisites
  • Install and configure Windows Compute Cluster 2003 as described in previous section. All services will be configured on public network, because each machine has only one network card. Let the cluster to have the following configuration:
    • Machine 1: HEADCLUSTER210 – the head node of the cluster, it is not a computational node.
    • Machine 2: CLUSTERNODE2101 – first computational node in the cluster.
    • Machine 3: CLUSTERNODE2102 – the second computational node in the cluster.
  • Check that the cluster is configured and running correctly, using Compute Cluster Administrator interface.
  • Install Microsoft Compute Cluster Pack SDK x64 version on each node, in the same location. Because CCS 2003 runs on a x64 architecture all application and processes that runs on the cluster will be x64 platform based.
  • Check that MPI services are up and running on all nodes.
  • Install Visual Studio 2008 Professional Edition or Visual Studio 2008 Team Edition on the computational nodes – CLUSTERNODE2101 and CLUSTERNODE2102 in the same location on EACH  node. Be sure that you install the extensions necessary for building x64 platform based applications. 

                           Check x64 extenstions 

  • Check that Visual Studio Remote Debugger (mpishim.exe) is installed on all nodes in the same location.
  • Modify the registry:

Cmd.exe has an issue with UNC paths.  MPI Debugging relies on these paths so just to be safe and make sure nothing breaks, carry out the following modification on each of the clusters.  Access the following registry key:

HKEY_CURRENT_USER\Software\Microsoft\Command Processor

Add a DWORD entry entitled “DisableUNCCheck” and set the value to 1:

Modify the registry

2.Running the application
  2.1.Configure a job with the Job Scheduler

If you want to have something done at the cluster for you, then you need to use the job scheduler. Be sure that you are logged in on the workstations using a domain account with administrative rights. All jobs and will have to be submitted using this account.  Debugging is no exception, as you need to create an empty job that will host the debugging application.To get started, open the Job Scheduler->File menu->Submit Job:                                                                     

Create a job on the cluster

Name the job “Debug Job” and move over to the Processors tab.  Select the number of processors you would like to use for this job and then, check the box that says “Run Job until end of run time or until cancelled”.   Failure to check this box will cause the empty job to run and finish. The job must to continually run, so that Visual Studio will then attach the running processes to this specific job.

  Select job’s running time

Move to the Tasks and add to the tasks list msvsmon.exe in order for the Visual Studio to communicate with the Visual Studio Debugger when running the parallel application.

 Add msvsmon.exe to the tasks list

Move to the Advanced tab and select which nodes will be part of your debugging scheme. In our case we will use only the twos nodes we have, but other computational nodes may be added.

 Allocate nodes for the job

Click on submit job, and the  job has to run. Write down the ID of the job (in this case, it is 16) as it will be used further.

 Running created job

   2.2.Configure Visual Studio
On CLUSTERNODE2101:
  • Open Visual Studio 2008 and create new application. See previous section “Debugging parallel applications with Visual Studio 2008″ for more detail about creating a new application.
  • Let for example the parallel value calculation of PI:

#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include “mpi.h”
int main(int argc, char *argv[])
{
      int         NumIntervals      = 0;  //num intervals in the domain [0,1] of F(x)= 4 / (1 + x*x)
      double      IntervalWidth     = 0.0;      //width of intervals
      double  IntervalLength  = 0.0;      //length of intervals
      double      IntrvlMidPoint    = 0.0;      //x mid point of interval
      int         Interval          = 0;  //loop counter
      int         done              = 0;  //flag
      double      MyPI              = 0.0;      //storage for PI approximation results
      double      ReferencePI       = 3.141592653589793238462643; //ref value of PI for comparison
      double  PI;
      char  processor_name[MPI_MAX_PROCESSOR_NAME];
      char  (*all_proc_names)[MPI_MAX_PROCESSOR_NAME];
      int         numprocs;
      int         MyID;
      int         namelen;
      int         proc = 0;
      MPI_Init(&argc,&argv);
      MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
      MPI_Comm_rank(MPI_COMM_WORLD,&MyID);
      MPI_Get_processor_name(processor_name,&namelen);

      all_proc_names = malloc(numprocs * MPI_MAX_PROCESSOR_NAME);

      MPI_Gather(processor_name, MPI_MAX_PROCESSOR_NAME, MPI_CHAR, all_proc_names, MPI_MAX_PROCESSOR_NAME, MPI_CHAR, 0, MPI_COMM_WORLD);

      for (proc=0; proc < numprocs; ++proc)
            printf(“Process %d on %s\n”, proc, all_proc_names[proc]);

      while (!done) //loops until done == 0
      {
            IntervalLength = 0.0;
            if (MyID == 0){
                  printf(“\nEnter the number of intervals: (0 quits) “);
                  fflush(stdout);
                  scanf_s(“%d”,&NumIntervals);
                  //NumIntervals = 10;
            }
            MPI_Bcast(&NumIntervals, 1, MPI_INT, 0, MPI_COMM_WORLD);   /* send number of intervals to all procs */
            if (NumIntervals == 0)
                  done = 1;   //exit if number of intervals = 0 
            else
            {
                 //approximate the value of PI
                  IntervalWidth   = 1.0 / (double) NumIntervals;          
                  for (Interval = MyID + 1; Interval <= NumIntervals; Interval += numprocs){
                        IntrvlMidPoint = IntervalWidth * ((double)Interval – 0.5);
                        IntervalLength += (4.0 / (1.0 + IntrvlMidPoint*IntrvlMidPoint));
                  }
                  MyPI = IntervalWidth * IntervalLength;
                  MPI_Reduce(&MyPI, &PI, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);

                  //report approximation
                  if (MyID == 0){
                  printf(“PI is approximately %.16f, Error is %.16f\n”,
                              PI, fabs(PI – ReferencePI));
                  //done=1;
                  }
            }
      }
      MPI_Finalize();
      //printf(“Hello world”);
}

  • Go to Project->Properties (or Alt + F7).
  • In Configuration Properties tab, select Debugging under the General tab.
  • Set Debugger to Launch to property value to MPI Cluster Debugger. The next screen shot show the values of the debugger properties:

Project settings 

MPI Run Command:  mpiexec. This is required for MPI application.
MPIRun Arguments:  The first argument “-job 3.0″ is to specify which is the job in the scheduler to use.  In my case, it was 3 when I created the job, and the 0 is to specify the task, which every job has by default.   We then have “-np 2″ which is used to specify that we will be using 2 nodes for this job.  Finally you see there is “-machinefile \\kim03a\bin\machines.txt“.  The “-machinefile” is used to specfify the UNV location of a text file that contains the names of the machines that will be part of this job.  The text file should have the names of the nodes on each line. The first line should contain the machine on which is made the application debugging in order not to redirect standard console input/output. So on CLUSTERNODE2101 machines.txt will look like this:

Machine file 

The name of the machines were duplicated because the physical machines on which the cluster is deployed are dual core, and the job that runs on the cluster needs to know a name for each core.

Application Command: This is the UNV path to the MPI application that you would like to debug.  This application HAS to be compiled to 64-bit and debugging symbols should be in that same directory as well.
MPIShim Location: In this location, it is specified the path to the x64 mpishim.exe binary. Mpishim should exist on each and every one of the machines at the specified local path.
MPI network security mode: Accept connections from any address to avoid problems.

  • In order to run C code and not C++ code you have to set up the compiler to compile the code as C code and not as C++ code. For this go to C/C++ -> Advanced tab and set the value of Compile as property to Compile as C++ or Compile as C depending on the written code. For the examples presented on the MPI’s API site you need to compile the project using Compile as C.

 Set compilator for C code

  • Select Linker->General tab and set the value for Additional Library Directories to “C:\Program Files\Microsoft Compute Cluster Pack\Lib\amd64” .The path depends on the place where CCP SDK was installed on the local machine.

Add additional library directories

  • Go to Linker->Input tab and add msmpi.lib to Additional Dependencies.

Add additional msmpi.lib

  • Go to C/C++->General tab and set the value for Additional Include Directories to “C:\Program Files\Microsoft Compute Cluster Pack\Include”.

 Include additional directories

  • Select Configuration Manager in order to selectx64 platform. Choose New and select x64 platform.

 Modify application platform

  • Set the Post Build Action. The executable should be copied in the shared location on the cluster in order to be accessible to all nodes in the cluster. In this case the shared location is headcluster210\PDC

Set post build action

  • Go to Tools->Options->Debugging tab and uncheck Break all processes when one process break.
  • Build the application and resolve any kind of errors.
  • Start debugging the application. From Debug->Windows->Processes you can see the processes that currently are running.
  • The result looks like this:

 Running parallel application 

The MPI application is running parallel on the cluster. In this case there are 4 processes that are running parallel, two of them on a compute node (node 1) and the other two on the other compute node (node 2).

On CLUSTERNODE2102:
  • Copy the application created recently on the second node and the settings too
  • Modify machines.txt (or create another file). Now it looks like this:

 Machine file 

  • Build the application and resolve any kind of errors.
  • Start debugging the application. From Debug->Windows->Processes you can see the processes that currently are running.
  • The result looks like this:

 Running parallel application 

The MPI application is running parallel on the cluster. In this case there are only 3 processes that are running parallel (-np 4 was modified to -np 3), two of them on a compute node (node 2) and the other one on the other compute node (node 1).

Some common problems when debugging MPI application on a cluster and their solutions can be found here.

CCS 2003 & MS-MPI (part 1)

Filed under: Parallel programming — Tudor Cret @ 6:11 pm

Around the world the computation power increseas every day. More power is needed because the amount of data increseas exponentially. Sometimes a single computer is not enough to complete a task or a job in a decent amount of time. In the fight with the execution time we created so called super-computers and we started to write parallel applications.

Microsoft has a solution for creating a super-computer: Windows Compute Cluster Server 2003, a high performance computing cluster consisting of a head node and one or more compute nodes. CCS uses MPI standard ( MS-MPI is a version of MPI implemented by Microsoft) to run parallel applications on the cluster. It may be tricky sometimes to create parallel applications and to run them on a cluster. In the next sections I will show how to create and to run such an application on a CCS 2003 cluster. First of all let’s install and configure Windows Compute Cluster 2003.

Software requirements

Supported operating systems for head node and compute nodes:

  • Microsoft Windows Server 2003, Compute Cluster Edition
  • Microsoft Windows Server 2003, Standard x64 Edition
  • Microsoft Windows Server 2003, Enterprise x64 Edition
  • Microsoft Windows Server 2003 R2, Standard x64 Edition
  • Microsoft Windows Server 2003 R2, Enterprise x64 Edition

An overview of Windows Server 2003 x64 Editions is available here.

In addition, Windows Compute Cluster Server 2003 also requires the following:

  • Microsoft .NET Framework 2.0
  • Microsoft Management Console(MMC) 3.0 Pre-Release
  • Microsoft SQL Server 2000 Desktop Engine (MSDE) to store all job information

Network requirements

CCS 2003 supports five different cluster topologies. Each topology has implications for performance and accessibility. The topologies involve at least one and possibly three different networks: public, private, and Message Passing Interface (MPI).

  • 1. Two network adapters on the head node, one network adapter on compute node
  • 2. Two network adapters on each node
  • 3. Three network adapters on the head node, two on compute nodes
  • 4. Three network adapters on each node
  • 5. One Network Adapter Per Node

More details about system requirements and network topologies can be found here.

Install and configure CCS

Checklist used to install and configure CCS 2003:

  1. Review prerequisites and assure that the computers used to configure as cluster nodes meet the all preconditions.
  2. Create a cluster head node.
    1. Install a supported x64 edition of Windows Server.
    2. [optional]Create a second partition in order to use RIS and compute nodes automated adding.
    3. Join the head node to an Active Directory domain or use Dcpromo.exe to create one and to make the node a domain controller.
    4. Configure DNS, DHCP and other required network services for the cluster’s network.
  3. Configure the cluster head node.
    1. Start the Compute Cluster Pack Installation Wizard from the Computer Cluster Pack CD.
    2. Select “Create a new compute cluster with this server as the head node” and click next.
    3. Define the network topology. Use To Do List that is displayed in the Compute Cluster Administrator.
    4. Disable or enable firewall on the public network. Use To Do List that is displayed in the Compute Cluster Administrator.
    5. [optional] Configure RIS and RIS installation images for compute nodes.
  4. Add compute nodes to the cluster.
    1. Verify prerequisites for the machine that is going to become a compute node.
    2. Install a supported x64 edition of Windows Server.
    3. Add the compute node to the same Active Directory domain with the head node.
    4. Add the compute node to the cluster. Use the Compute Cluster Pack CD and select join this server to an existing compute cluster as node for manual addition.

More information about deploying CCS 2003 can be found here.

  Now that we have installed and configured the cluster we will see more about MPI and parallel application in part2 of this post.

« Previous Page

Create a free website or blog at WordPress.com.