Output to command-line if started from command line

Question

I'm writing an application that can be started either as a standard WinForms app or in unattended mode from the command-line. The application was built using the VS 2k5 standard WinForms template.

When the application is executed from the command-line, I want it to output information that can be captured by the script executing the application. When I do this directly from Console.WriteLine(), the output does not appear, although it can be captured by piping to a file.

On the other hand, I can force the application to pop up a second console by doing a P/Invoke on AllocConsole() from kernel32. This is not what I want, though. I want the output to appear in the same window the application was called from.

This is the salient code that allows me to pop up a console from the command line:

<STAThread()> Public Shared Sub Main()

    If My.Application.CommandLineArgs.Count = 0 Then
        Dim frm As New ISECMMParamUtilForm()
        frm.ShowDialog()
    Else
        Try
            ConsoleControl.AllocConsole()
            Dim exMan As New UnattendedExecutionManager(ConvertArgs())
            IsInConsoleMode = True
            OutputMessage("Application started.")
            If Not exMan.SetSettings() Then
                OutputMessage("Execution failed.")
            End If
        Catch ex As Exception
            Console.WriteLine(ex.ToString())
        Finally
            ConsoleControl.FreeConsole()
        End Try

    End If

End Sub

Public Shared Sub OutputMessage(ByVal msg As String, Optional ByVal isError As Boolean = False)
    Trace.WriteLine(msg)
    If IsInConsoleMode Then
        Console.WriteLine(msg)
    End If

    If isError Then
        EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Error)
    Else
        EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Information)
    End If

End Sub

Answer 1

Raymond Chen recently posted (a month after the question was posted here on SO) a short article about this:

How do I write a program that can be run either as a console or a GUI application?

You can't, but you can try to fake it.

Each PE application contains a field in its header that specifies which subsystem it was designed to run under. You can say IMAGE_SUBSYSTEM_WINDOWS_GUI to mark yourself as a Windows GUI application, or you can say IMAGE_SUBSYSTEM_WINDOWS_CUI to say that you are a console application. If you are GUI application, then the program will run without a console.

The subsystem determines how the kernel prepares the execution environment for the program. If the program is marked as running in the console subsystem, then the kernel will connect the program's console to the console of its parent, creating a new console if the parent doesn't have a console. (This is an incomplete description, but the details aren't relevant to the discussion.) On the other hand, if the program is marked as running as a GUI application, then the kernel will run the program without any console at all.

In that article he points to another by Junfeng Zhang that discusses how a couple of programs (Visual Studio and ildasm) implement this behavior:

How to make an application as both GUI and Console application?

In VisualStudio case, there are actually two binaries: devenv.com and devenv.exe. Devenv.com is a Console app. Devenv.exe is a GUI app. When you type devenv, because of the Win32 probing rule, devenv.com is executed. If there is no input, devenv.com launches devenv.exe, and exits itself. If there are inputs, devenv.com handles them as normal Console app.

In ildasm case, there is only one binary: ildasm.exe. It is first compiled as a GUI application. Later editbin.exe is used to mark it as console subsystem. In its main method it determines if it needs to be run as console mode or GUI mode. If need to run as GUI mode, it relaunches itself as a GUI app.

In the comments to Raymond Chen's article, laonianren has this to add to Junfeng Zhang's brief description of how Visual Studio works:

devenv.com is a general purpose console-mode stub application. When it runs it creates three pipes to redirect the console's stdin, stdout and stderr. It then finds its own name (usually devenv.com), replaces the ".com" with ".exe" and launches the new app (i.e. devenv.exe) using the read end of the stdin pipe and the write ends of the stdout and stderr pipes as the standard handles. Then it just sits and waits for devenv.exe to exit and copies data between the console and the pipes.

Thus even though devenv.exe is a gui app it can read and write the "parent" console using its standard handles.

And you could use devenv.com yourself for myapp.exe by renaming it to myapp.com. But you can't in practise because it belongs to MS.

Answer 2

If you want to check if your app is started from the command line in .NET, you can use Console.GetCursorPosition(). The reason that this works is that when you start it from the command line, the cursor moves away from the initial point ((0, 0)) because you typed something in the terminal (the name of the app). You can do this with an equality check (code in C#):

class Program
{
    public static void Main
    {
        if (Console.GetCursorPosition() == (0, 0))
        {
            //something GUI
        }
        else
        {
            //something not GUI
        }
    }
}

Note: You must set the output type to Console Application as other output types will make Console.GetCursorPosition() throw an exception.