Course: Complete Guide to XML for Microsoft Developers
Neal Walters published his new XML for Microsoft developers training course on Udemy as of June 2020. Parts of the course apply to non-Microsoft environments and that includes XML basics, XSD, XPath, XQuery, and XSLT, which are all W3C (W3.org) standards. The course shows how to deal with XML in three programming languages: 1) Visual Studio, 2) PowerShell, and 3) Transact SQL (T-SQL) for SQL Server. The course includes other tools such as how to check, format, and validate XML with the NotePad++ plugins, and how to use other tools like VSCode and Microsoft's XML Notepad. The full outline is below.
Intro to XML
- XML Files with Elements and Nesting (Flight01/Flight02) and Empty Element
- XML Files with Attributes (Flight03-07)
- XML Files - Misc Features - Declarative, Comments (Sample01)
- XML Encoding Special Characters (Sample02) - Intro to PowerShell
- XML Unicode & Byte Order Mark (Sample03_Unicode.xml)
- XML Files - Namespace
- Review with chart (see image tab)
- XML to JSON using PowerShell/NewtonSoft .Net Library
- XML to JSON - PowerShell - formatting/indenting
- Finding Well-Formed Errors (The Divide and Conquer Method)
XML Tools: Editors & Browsers
- Editing XML with NotePad++
- Editing XML with Visual Studio (2019)
- Adding Xpath Tools (Extension) to Visual Studio (From Marketplace)
- Editing XML with VS Code
- Microsoft XML Notepad
- Commercial Tools: Stylus Studio, XML Spy, Oxygen
- How Browsers handle XML (normal, with errors, with stylesheet)
- XPath Intro - and How to Practice Answers to Quizes In this Module
- Simple element selection (including subscripts and text())
- Attributes and simple "where" clauses
- More where clauses (elements and/or)
- Relative paths
- XPath Axes - Documentation and Samples
- Namespaces - Part 1
- Namespaces - Part 2
- Real world, Difficult XPath
- Functions: Count, Sum, Contains, Starts-with, Substring-Before
- XPath 2.0, 3.0, 3.1
C# Programming - XmlDocument and XPath
- XMLDocument - Load, LoadXML, Save, PrettyPrint
- XMLDocument, SelectNodes, SelectSingleNode, GetElementsByTagName
- XMLDocument - pretty print to a string
- XMLDocument - Navigation: NextSibling, PreviousSibling, ParentNode
- Validate XML data against a Schema
- Add a new attribute to existing element
- Add a new element to existing element
- Removing elements and attributes (XmlNode Properties/Methods)
- C# Assignment - Getting values from an XML Config File
- C# Assignment Solution - Getting values from an XML Config File
C# - Streaming - XMLReader
- Validate XML data against a Schema
- XPathNavigator (Streaming)
- Saxon (Third Party Library)
- Microsoft CORE
- XSD Schema Intro (also mention DTD, XDR Schemas, RelaxNG, Schematron)
- Generate Schema from Web - Validation in NotePad++
- PowerShell - using .Net Library - Schema Inference
- XSD Elements and Attributes (MaxOccurs)
- XSD Restrictions
- XSD Empty, Elements Only, Text Only, Mixed
- XSD Indicators, All, Choice, Sequence, maxOccurs, MinOccurs, Groups
- Any Element and anyAttribute
- XSD Data Types
- Allowing Nulls
- Validating the Schema Syntax in C#
- PowerShell Equivalent of Last Video
- Discussion of Schema Editors (And Demo of BizTalk)
- Industry Standard Schemas - Example OAGIS
- Schema/Contract Development
- Generate JSON Schema from JSON file
- How Mapping Fits into the Business Processes
- Intro to XSLT - Big Picture
- HelloWorld Example - Running .NET XSLT 1.0 in PowerShell
- HelloWorld Example - Running Saxon XSLT 3.0 in PowerShell
- ForEach and ApplyTemplates
- If/Test and Choose/When (Long Version with Errors/Debugging)
- If/Test and Choose/When (Short pre-canned version)
- Variables in XSLT
- Key and Xpath Lookup
- Muenchian Grouping in 1.0 and new feature in 3.0
- Assignment 1 - Map Shakespeare to New XML
- Assignment 1 - Solution
- Assignment 2 - Build HTML Web Page from Flight03
- Assignment 2 - Solution - Part 1 of 3
- Assignment 2 - Solution - Part 2 of 3
- Assignment 2 - Solution - Part 2 of 3
- Introduction (Slide Presentation)
- Saxon.exe - Example Flight03.xml
- Saxon.exe - Example Shakespeare Hamlet.xml
- Saxon.exe - Example Vidoes.xml - join and multiple FLWOR
- XQuery in SQL Server Queries
- PowerShell Examples (Saxonica and SQL/XQuery)
C# Programming with Saxonica API
- Downloading and Documentation
C# Programming and XSD.exe - Serialization/Deserialization
- Serialize (Object to XML) and Deserialize (XML to Object)
- XSD.exe - big picture (slides)
- XSD.exe - Generated a class from a schema
- Creating a fully populated object from the generated class
- XSD.exe - Generated schema from a class
XML DataType in SQL Server
- The XML Column
- Populating XML Column with Shakespeare File Data
- Querying XML Column using XPath
- Querying XML Column using XPath
- Query XML Column - one to many - with Cross Apply
- XML Column and the Like Clause
- XML Column and the Exist Method
- XML Column and the Modify Method
- Working with Namespaces Prefixes
- Intro to XML Column Indexes
- Vote for Future SQL Improvements
Loading XML into SQL Tables (using T-SQL and SSMS)
- Bonus - Designing/Creating Tables - Part 1
- Bonus - Designing/Creating Tables - Part 2
- Bonus - Database Redesign - Part 3
- Bonus - Manual Data Entry Airlines, Airports, Flights
- Populate Reservation Table from Flight03.xml using OpenRowSet
- Improve Last SQL Script (for Reservation Table)
- Manually Entering Flight Data
- Loading FlightReservation Table (one to many relationships)
- Canned shorter version of prior Video
- Bonus: Overview of UpdateGrams (SQLXML 4.0)
- Bonus: How The Database could be Populated with Biztalk (and UDTs)
- Assignment - C# Load of Database
- Assignment - C# Load of Database Solution
Querying SQL Data as XML
- Intro to the "For XML" clause and it's variations
- For XML RAW in T-SQL
- For XML AUTO in T-SQL
- For XML EXPLICIT in T-SQL - Introductory Concepts
- For XML EXPLICIT in T-SQL - Sample
- For XML PATH in T-SQL - Introductory Concepts
- For XML PATH in T-SQL - Sample
- PowerShell to run For XML SQL command and save result to file
XML and PowerShell
- Simple PowerShell with Flight01.xml
- Some PowerShell Tips for Newbies (MOVE TO INTRO???)
- Simple PowerShell - Pipe to Format-Table and Format-List
- Using Select-XML or the .NET SelectNode/SelectSingleNode
- Convert XML To CSV or to HTML
- Convert To XML
- Converting XML to JSON
- Pretty-Print all XML in disk directories (preserving datetime)
SFTP/OpenSSH Course (including BizTalk SFTP Adapter)
Do you need to learn how run an SFTP server? OpenSSH is the free/open-source tool used by many companies. It runs on both Windows And Linux servers. Neal released his second Udemy course in May of 2020, titled SFTP Client/Server Deep Dive (with OpenSSH & WinSCP). The course was designed for system administrator and/or developers that need to install and support SFTP (Secure/SSH File Transport Protocol). This if often needed in B2B scenarios where your company needs to send or receive files to various trading partners, customers, or vendors. It is common to transfer XML, EDI (Electronic Data Interchange), or CSV files to and from SFTP sites. The original FTP protocol goes back many years, but it was never secure. The data and even the passwords were sent in clear text over the internet. FTPS was one solution to that issue, and it uses SSL/TSL certificates to perform encryption/decryption. But SFTP is one of the preferred methods to use in today's world, and OpenSSH is probably the most popular software package for running an SFTP server. While SSH can work with just a user/password, it is common for the trading partner to send you his private key, and you need to know how to install that on your OpenSSH server. OpenSSH also generates your own private/public key-pairs when you install it. Neal frequently refers to the concept of "TOFU" - "Trust On First Use". In this course, Neal explains the difference between FTP, FPTS, and SFTP, what he calls "FTP Soup". He also explains SSH (Secure Shell), the protocol on which SFTP runs. He covers the most common clients, such as WinSCP and FileZilla. Another part of the course is how to use SFTP in the real world, The course shows how to schedule SFTP transfers at a given time (daily, weekly, etc...). For Windows, this is done on Windows task manager, and on Linux, it's done by a "CRON JOB". In either case, the developer or administrator might have to write a small script to do the transfer (BASH for Linux, and PowerShell or a .bat/cmd file for Windows). If a developer is using C#, it is possible to send and receive SFTP file directly from that .NET program. WinSCP includes a .NET library. As a matter of fact, this library is so solid, that Microsoft BizTalk 2016 decided to use it instead of their own internally written SFTP code that was used in BizTalk 2016. Neal also demonstrates how to use Microsoft BizTalk Server to send and receive SFTP files. BizTalk is a popular middleware, business process manager software platform used by many Windows customers for Business to Business (B2B) and Enterprise Application Integration (EAI). Beyond SFTP, there are higher level enterprise solutions called Managed File Transfer. They often contains a web portal to audit and view all the transfers, and all the logging is done to a typical SQL database. For those students that want to use or learn Unix, Neal shows how to install Ubunutu 20.04 using Oracle Virtual Box. He then installs OpenSSH on that server, then shows how to configure it for multiple users. The entire course consists of 61 "lectures" (mostly hands-on demonstration videos, and a few slide presentations). There are overs 6 hours videos in total. Starting learning SFTP and OpenSSH today.
Real World BizTalk 2020 Developer Course
Neal Walters published his new BizTalk 2020 training course on Udemy as of May 2020.
Microsoft BizTalk Server is an integration software and framework for building B2B (Business to Business) and performing EAI (Enterprise Application Integration).
Most companies have various vendors, customers, and trading partners where they need to securely exchange and process data. BizTalk helps automate those business processes (often call BPA - Business Process Automation).
The course actually covers common features of BizTalk that would be the same from BizTalk 2010, 2013, 2016, and 2020. BizTalk 2020 was just released in February of 2020. Neal's last BizTalk coures were created back for the BizTalk 2006 release, and he has put them on his YouTube channel called "MrBizTalk".
The courses beings with an introduction that explains "What is BizTalk?" and why companies use it, what is its market share, and what type of job roles are there in the BizTalk space. The foundational architecture is laid out, examining the BizTalk Architecture poster, explaining applications, hosts and host instances, adapters, schemas, maps, orchestrations, pipelines, business rule engine, and the ESB Toolkit.
Quizes are included at the end of each Sections. They grades don't matter, but the purpose is review the most important concepts covered in the Section. Section 3 - gives a great overview of Content-Based routing, using the PassThrough Pipeline. The student doesn't need to understand schemas and maps at this point. Neal sends even .pdfs and .jpgs through BizTalk without ever opening Visual Studio. This also provides a nice gentle introduction to the BizTalk Admin Console, and a feature called "Message Tracking" which allows the administrator or developer to see a log of what has processed through the system.
This section also includes lesson on how to set up subscriptions and even how to have multiple subscribers to the same message.
Section 4 - Again using the easy-does-it-approach, Neal demonstrates the creation of two schemas and a map, and shows how BizTalk shines when it comes to simple mapping. With a zest for clever-naming, BizTalk created something called "Functoids" which developers can place on the mapping grid and perform manipulations of the data as it gets mapped.
Neal never shies away from showing the most common errors that developers can get, even in simple file drops and mapping. One thing that happens to almost all developers at one time or another is that the same schema is put into two different .NET assemblies and deployed. Neal shows the odd error and how to get around it. This is an great opportunity to teach about the BizTalk message-type which is the Namespace followed by a pound sign and the root element.
Section 5 - introduces the concept of an orchestration to process the message. It calls a Business Rule as well.
Section 6 - Introduces user-created (aka "custom") pipelines. Rather than spending an hour or two learning the ins and outs of developing a custom pipeline component in C#, the course just downloads a free pipeline component (the Eliasen RegEx/String Replace component). The course shows how to compile and install it into the GAC, then how to use it a custom pipeline. It's very useful for doing a string-replace on data inside the incoming message.
Section 7 - Now that the big picture has been accomplished, the student needs to learn all the other powers and tricks of XSD Schemas. This is the perfect time to teach XPath as well, as BizTalk uses XPath in many different ways (promoted fields, distinguished fields, XPath statement in an orchestration, Business Rules, and so on). One of the not so obvious features of schemas is the ability to have an envelope schema which allows a hierachical message to be debatched.
Section 8 - And now that schema have been covered "in depth", it's time to cover "Maps in Depth". There are many functoids to discuss and demonstrate, including the table looping and scripting functoids. The first allows for some powerful mapping when the schemas have a different "shape", and the second allows you to call C# methods from the map, or even include XSLT fragments inline. This section also covers how to convert CSV data to XML and vice verse using the "Flat File Pipeline".
Section 9 is basically the module on Extensible Stylesheet Language Transformation (XSLT). All maps actually get converted to XSLT, but BizTalk allows the developer to write the entire map in XSLT if he or she chooses. The first lesson converts an existing functoid-based map to XSLT, making it more readable and efficient over the next few lessons.
This lesson also covers XSLT debugging and a new feature only available in BizTalk 2020, the ability to use XSLT 3.0 in maps via the Saxon parser (by Saxonica). The course shows how to download the Saxonica parser and use it to test the XSLT code before using it in BizTalk.
Section 10 - All BizTalk developers and/or administrators need to know how to migrate or transfer applications from one environment to another (for example, Development to QA, then QA to Production). This is typically called migration or deployment. This section covers the ways to do it using the graphical interface inside the Biztalk Admin Console, and also by using the BTSTask command.
Section 11 goes into a lot more depth on Orchestrations. First, how to use use promoted and distinguished fields, and/or XPath. Section 11 also shows multiple ways to debug and trace an orchestration. Many students want to know how to name a file coming out of an orchestration, and that is taught as well.
Section 12 teaches some more advanced features called "Convoys", which require the use of correlation. The two most common convoys are the "sequential convoy" and the "parallel" convoy. The sequential convoy waits for either a new message to arrive (in a loop), or a certain amount of time to pass (via the Delay shape). The sequential convoy is used to implement an aggregation pattern, where multiple messages come into the orchestration, and then are combine into one new larger XML document which is output.
The course has over 21 hours of video training. Rather than overwhelm the student by including even more in that course, Neal hopes to release additional courses that will cover speciality topics like WCF-SQL, WCF web services (SOAP and Rest).
Neal has already created a second course that is in review now. It covers how to setup an SFTP Server using OpenSSH or a commercially available product called CompleteFTP. Neal used CompleteFTP in a prior company, and it handles FTP, FTPS and SFTP. CompleteFTP is a very reasonably priced product, but OpenSSH is open source (i.e. free). However, it takes more knowledge on how to use and configure it. Neal shows how to run OpenSSH on a Windows or Unix (Ubuntu) server. Those interested in saving more money can do so by using Unix, as long as their staff can support it. This second class of course covers how to use the SFTP adapter that is available in BizTalk 2020. It's base don the WinSCP .net Library.