Tag Cloud

Recent Posts

Topics

Archives

Top Posts & Pages

Links:

Recent Post Comments

Administration

How To Create a Networked Git Repository on Windows From Scratch (in less than 60 seconds)

Seekers,

Everything I found out there to do this was either wrong, had typos, or said it was for a ‘Remote’ (networked) repository, but they were talking about Github, or something else… I wanted at least one post on the Internet to be accurate, and do what it said it was going to do.

Here we just create a fresh, empty repository on a file share (share location: D:\GitRepos\testRepo, local directory will be C:\Users\Pat\testRepo (filename created when checked out)):

C:\Users\Pat>d:

D:\>cd GitRepos

D:\GitRepos>mkdir testRepo

D:\GitRepos>cd testRepo

D:\GitRepos\testRepo>git init
Initialized empty Git repository in D:/GitRepos/testRepo/.git/

D:\GitRepos\testRepo>touch NewRepository

D:\GitRepos\testRepo>git add *

D:\GitRepos\testRepo>git commit -a -m “First Commit…”
[master (root-commit) 45fac89] First Commit…
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 NewRepository

Next we go back to the directory we want the repo checked out to:

D:\GitRepos\testRepo>c:

C:\Users\Pat>git clone //doozy/gitrepos/testRepo
Cloning into ‘testRepo’…
done.

note: In the above, even though my repo is on the same box as my local checkout, I am showing here the way other users, and myself, can get to it to show you it is, in fact, a network shared repository.

C:\Users\Pat>ls
[...]
testRepo
[...]

C:\Users\Pat>ls -l testRepo
total 0
-rw-rw-rw-  1 Pat 0 0 2014-04-11 17:59 NewRepository <— the file we ‘touched’ earlier

next we will create a branch off of our master, calling it ‘development’. This is the one we will primarily work with, and we will branch/merge to/from this branch to create releases:

C:\Users\Pat>cd testRepo

C:\Users\Pat\testRepo>git branch development

C:\Users\Pat\testRepo>git checkout development
Switched to branch ‘development’

C:\Users\Pat\testRepo>touch inDevelopmentBranch

C:\Users\Pat\testRepo>git commit -a
On branch development
Untracked files:
inDevelopmentBranch <— the file we just created is in the local working dir, but just like in SVN, it has not been ‘added’ to version control

nothing added to commit but untracked files present

C:\Users\Pat\testRepo>git add inDevelopmentBranch

note: You can also do: “git add *” to add all, or any other wildcards…

C:\Users\Pat\testRepo>git commit -a
[development 32a74f0] This is the first commit for the development branch… testRepo is the repository…
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 inDevelopmentBranch

C:\Users\Pat\testRepo>git –version
git version 1.9.0.msysgit.0

Done!

What Makes A Good Presenter?

Wendy Russell (the above is not hers) covers most of the mistakes we can make if we lack experience or plain common sense. I’ll comment on her bullets:

I Thought You Were Bringing The Extra Lightbulb

I have never had to bring a projector to present to a large audience. I leave all that up to their IT/AV people, and if all else fails, I always start on time speaking to the audience directly while it is remedied, instead of the the ultimate faux pas of turning your back on your audience.

Information Underload

If you simply memorize, not only does it show, but you are ‘deer in the headlights’ when there is a question. You must be the MASTER of your material, such that you can give the info in any order, with any media, while still ‘working the audience’.

What’s It All About?

“This is the opposite of Information Underload. You know so much about the topic, that you jump from here to there and back again talking about everything there is to know about your brand new widget, and no one can follow the thread of the presentation.

Corrected Presentation Technique #3
Use the K.I.S.S. principle (Keep It Simple Silly) when designing a presentation. Stick to three, or at the most, four points about your topic and expound on them. The audience will be more likely to retain the information.”

When You Don’t Have Content, Dazzle Them With Complicated Diagrams (Ignorance)

Oh, the stories I could tell you…

Did You Bring Your Glasses?

Small fonts kill it! If you can’t say it in 3 bullets, then you don’t understand what you are talking about. Try to remember that you are not consulting or teaching!

Divine Design

A whole topic here I could spend hours on… because it took me years to master. What seems like high contrast on your LCD screen will not look the same on a projector-especially one that is not 100% color matched to your design screen (and none of them are the same color as your design screen).

What to do? Well, I have yet to see a projector that failed to make white look like white, and black look like black. You take it from there…

Do Not Animate.

Anything. Nuff said.

(note: some of my looping demos for booths are exceptions, but technically these aren’t the same as presentations)

There are more, such as body english, dealing with sharks, snipers, and elephants (the salesie types out there understand this), being too much sales ENGINEER, or too much SALES engineer, etc. but these are a few high points to think about…

As always, keep your horror stories coming!

pat
:)

Presentation Tips
More Presentation Tips

Adding And Removing Multi-Thread Producers and Consumers in VB.NET with BlockingCollection

Seekers,

This is what happens when you can’t find a solution anywhere on the Interwebs, and you just have to hack at it until you get it looking good [enough].

I have a situation where I need to parse objects coming in at high speed, but the parsing takes a relatively long time-far slower than they come in… I can tell how many cores are on a box pretty easily, and then I can carve up a team of worker threads (Tasks) to all divide & conquer! But the problem with ANY multi-threaded application is “How Many Threads?

There is no hard & fast answer, so don’t bother looking. My approach is to make a flexible approach that my main thread can monitor to see if throughput (aggregate work completed in X amount of time) is at maximum for the machine it happens to be running on. Also, that same machine may vary in load, RAM available, etc. over time, so you can’t just set it & forget it…

So this is the idea:

Tasks

So let’s see how I solved the issue of the blocking collection and then dishing our work to the Parser Tasks… I used a very generic, and easily re-usable approach of Producer and Consumer. In fact, in my example below, you can even adjust the amount of Producers as well. You can adjust my code below to more closely approximate your work, and then adjust the # of threads/tasks/Consumers to see how well parallelism can work for you.

First, the usual suspects in Imports..

The Producer(s) are initially set up with the above. While the itemsToProduce  doesn’t change during the program, the number of Producers & Consumers will. This is very rough as a draft, and it will no doubt be streamlined in your own code at some point, but this demonstrates how to solve this problem very well.

I used “IDs” to show in the output which thread was doing what. They only thing needed in production is the list of CTS instances:

The main machine here is that one little line:

Dim bc As New BlockingCollection(Of Integer)


Microsoft says:

BlockingCollection Overview
.NET Framework 4.5

BlockingCollection(Of T) is a thread-safe collection class that provides the following features:

  • An implementation of the Producer-Consumer pattern.
  • Concurrent adding and taking of items from multiple threads.
  • Optional maximum capacity.
  • Insertion and removal operations that block when collection is empty or full.
  • Insertion and removal “try” operations that do not block or that block up to a specified period of time.
  • Encapsulates any collection type that implements IProducerConsumerCollection(Of T)
  • Cancellation with cancellation tokens.
  • Two kinds of enumeration with foreach (For Each in Visual Basic):
  1. Read-only enumeration.
  2. Enumeration that removes items as they are enumerated.

itemId is just a variable that holds the fake payload. Producers will increment it by one to simulate a different object instance, or unit of work. You just change the type the BlockingCollection holds…

Now I’m not doing this in a FIFO way (which I will in production), but you can either do that, or even a FILO, as per Microsoft:


 

When you create a BlockingCollection(Of T) object, you can specify not only the bounded capacity but also the type of collection to use. For example, you could specify a ConcurrentQueue(Of T) object for first in, first out (FIFO) behavior, or a ConcurrentStack(Of T) object for last in, first out (LIFO) behavior.


 

Now THAT is useful! Here in this demo, I did it all willy-nilly… But like I said, for my specific need, I need FIFO, as in the diagram at the top…

Later, you will see the functions and subroutines, but the real magic here is in 2 collections-one for Producers and one for Consumers:

The Magic: As each Task(thread) is either created or closed, the corresponding CancellationTokenSource is either added or removed from the appropriate collection above.

Seriously, that is it! :)

Next in the code, the initial producers and consumers are created:

Aside from a few Thread.Sleep() calls, all the next part does is add or remove producer and consumer Tasks(threads). You can vary the initial values at the top to put it through the paces.

To create a Task… – CreateTask(<integer id for display>, <”Producer” or “Consumer”>)

To remove a Task, you (in one line) both get a random CTS, and then .Cancel() it:

GetRandomCTS(ProducerCTSs).Cancel()

GetRandomCTS(ConsumerCTSs).Cancel()

GetRandomCTS() takes the collection of CTS instances, picks one at random, then calls Cancel() on it.

And that is it!

Now for the fun parts:

Line 7: This is what we’ll be returning, a CancellationTokenSource

Line 16: ctsBag.ElementAt() allows us to pull out a specific CTS instance by number.

Below, the CreateTask takes an argument for the # you want it to display when running (just for demo, to see which thread is doing what), and a string to tell it whether you want a new Producer of Consumer. Sure, I could have made it more complex, but this is just a rough draft. :)

Line 7 & 10: These call the Producer() or Consumer() classes below, passing them the CancellationTokenSource needed to let them be able to elegantly be cancelled while running without corrupting any data.

t = Task.Factory.StartNew(Sub() Producer(taskId, token), token, TaskCreationOptions.LongRunning)

Did you notice TaskCreationOptions.LongRunning? That is good in my case, and it improves performance by telling the program to not worry so much about wating what happens for cancellations too closely.

How cool is that?!

So what does a Producer() class look like?

I know, I know… it looks complicated! But really it isn’t. I’m not that smart! 1/2 the code is just to catch & handle the cancellation requests so that the processing doesn’t corrupt any data. That, and a cheesy StopWatch() to time things… And yes, there are artifacts of earlier versions still commented out. Like I said “ROUGH“…

Line 17: Simply adds the itemId (our payload, could be anything) to the BlockingCollection (bc).

Line 20: If a cancellation, we take care of it here, and not a random part of the function, which would likely corrupt all kinds of things…

Line 31: I added this as a cheesy way to tell the Producers when to stop …producing. This variable (limit) is set at the top of the code.

Line 38: bc.CompleteAdding() – This is a signal to everyone using the bc (BlockingCollection) that there will be no more items added. This way, the consumers know when to stop …consuming!

“Why would they want to do that?”

Well, suppose you wanted a short-running task, or tasks, and needed to know they were done in order to continue…  Yes, in my case, they are long-running, and in production I’ll be starting each task with “TaskCreationOptions.LongRunning

The Consumer() class is almost identical, with just a few tiny differences:

Line 3: In both classes, we make sure to check right at the top to see if we’ve been cancelled. This way, we waste no time or resources if another Task/thread did the last piece of work just as we were being instantiated.

Line 13: itemOut = bc.Take() – Here we grab the next item (depends on FIFO or FILO/LIFO, as configured in the above discussion. This BlockingCollection does it all!

When you sit back and look at it, all the other code in this class is just to dress up Line 13!

So let’s fire this puppy up!

Was this the output you expected?

Grab the entire solution 7z’d up for you, at the link below…

Download Solution from HERE!

It took me a long time to figure out the whole CancellationToken concept, but now that I’m using it, and the bullet-proof-ness of the BlockingCollection, I’m confident my application can handle hundreds of objects per second without messing anything up.

My production application will read the amount of cores on the host machine, and use that to set the initial number of consumers. I will then adjust up & down, monitoring the time to complete (in an aggregate fashion), thus making the best of the host machine’s resources, understanding that the host machine may be doing many other things at the same time as my application.

Your thoughts, improvements, etc. appreciated!

pat
:)

“Insufficient resources to perform operation” With LOTS of Resources! WTF?

So there I was, programming away, when after only 300,000 messages (combined) in my queues, I got the dread “Insufficient resources…” message in my logs! This is an 8-Core, 4.3GHz, 32GB RAM box… CPU sits at 41°F to to 43°F and never breaks a sweat.. ever.

So WTF?

It didn’t take much looking to find that although you can configure no limits on each individual queue, Microsoft still imposes a total size limit on all queues on the box by default. “Whuh?” I know, right?!

Just right click on the Message Queuing icon/branch of the tree as below, and select Properties:

2014-03-24_Screenshot_207

Sure enough, there it is! Now I don’t care about the journal storage, as I’m not journaling. But the Limit message storage to (KB): is a limit on all queues combined even if you have unlimited selected for each and every queue!

So I unchecked them both (I have 3TB available in RAID1, so go for it!):

2014-03-24_Screenshot_209

So this cleaned things up IMMEDIATELY. Just to make sure, I clicked on the Storage tab, and went to the storage location to see how much stuff was in there:

2014-03-24_Screenshot_208

Well, what do you know! Now please note that my messages here are instances of a syslog message class that just as a single syslog message per instance. ~300,000 of these messages in the queue on disk taking up 1GB means that each one-although small-with XML overhead is taking up about 3.5KB each.

So when planning, remember it isn’t the actual size of the instance that will take up space, but all the metadata and XML overhead will add up to a LOT more than the size of your actual message.

pat
:)

How To Backup A Complete (rows, idx, pk) MS SQL 2012 Table In VB.NET

Experts,

I can’t find a cleaner way to do this… I tried to encapsulate all the logic into a single Function that returns a boolean on success/failure. Please orgive the roughness of the code!

NewTableName is just OldTableName_YYYMMDD …I could have used “SourceTableName”, but this is just the first hack at it… :)

 

I only have 3 source tables, so a Switch/Case is very practical in this early stage…

I’m still open to ideas, but I dug as far as I could, and I have to move on. Maybe I’ll re-visit it something more elegant comes along!

pat
:)