Created by: gwideman, Jan 31, 2012 9:33 pm
Revised by: gwideman, Aug 10, 2012 12:26 pm (19 revisions)

Overview

This article discusses Delphi's TWebBrowser, which is a wrapper for Microsoft's ShDocVW.dll WebBrowser control. In particular, the article pursues the topic of how to tame certain aspects of WebBrowser's behavior. This leads to a revised component, TGwWebBrowserTamed, which implements properties to disable features and behaviors which an application may want to avoid.
Delphi's TWebBrowser might be an attractive component for an application which needs to:
  • retrieve html pages from a web server
  • parse each page into a Document Object Model (DOM) tree, which the application can navigate to gather data of interest.
TWebBrowser, in unit ShDocVw, basically wraps the functionality provided by Internet Explorer (IWebBrowser2), notably including the parsing apparatus associated with MSHTML and its DOM.
The diagram to the right, borrowed from MSDN, sketches the architecture.MSDN-IE-Architecture.gif
This would appear to fulfill the requirements I set out above. However, the MS WebBrowser includes a whole lot of other behavior that our custom application may not want:
  • Downloading additional files that aren't needed, such as images, video, scripts, CSS and so on.
  • Running scripts (ie: Javascript and VBScript)
  • Playing video and audio
  • Throwing errors and/or putting up dialogs
  • Behaviors which vary, depending on the user's Internet settings.
These raise the question of how to disable or otherwise hem in the behaviors that aren't needed in a particular scenario.

Background

It would be convenient for component users if an application could control the issues listed above by manipulating some option settings exposed by TWebBrowser component, or even by navigating the object model into the underlying COM objects and setting some properties. Unfortunately, it's a bit more complicated than that.
The main issue is that, in the IE architecture, these issues are controlled by settings which the underlying web browser component must request from the containing application, settings referred to as "Ambient Properties" -- that is to say, properties supplied by the "ambient environment" surrounding the control.
The MS WebBrowser control architecture defines the interfaces through which the hosting application and the WebBrowser communicate. The WebBrowser implements a number of interfaces, and the host application or container implements the corresponding host ones. The main topic of of this article is the set of interfaces that the host should implement in order to supply the ambient property values.
Of special interest is the ambient property called "DLCONTROL", or "download control". This is a single integer whose individual bits control whether WebBrowser will automatically download files such as images, videos and scripts, and also whether WebBrowser will execute scripts of various types, and a variety of other behaviors.
Delphi's TWebBrowser, being a wrapper around MS WebBrowser, implements the basic apparatus for hosting the MS WebBrowser. However, it does not implement the ambient properties interface. The revised component described here will fill that gap.

What the DLCONTROL ambient properties control

The following table lists the functions of the individual bits within the DLCONTROL ambient variable expected by WebBrowser. This is essentially the same list provided by MSDN, with additional notes. The listing is in alphabetical order of constant name, not numerical order.
DLCTL_xxx constant
Description
DLCTL_BGSOUNDS
On = play doc's background sounds
DLCTL_DLIMAGES
On = Download images from the server.
DLCTL_DOWNLOADONLY
On = Download the page, but not display it.
DLCTL_FORCEOFFLINE
On = The browsing component will always operate in offline mode. This causes the BINDF_OFFLINEOPERATION flag to be set even if the computer is connected to the Internet when making requests through URLMON.
DLCTL_NO_BEHAVIORS
On = Do not execute any binary behaviors.
DLCTL_NO_CLIENTPULL
On = Do not perform any client pull operations. This relates to HTML META tag with attributes HTTP-EQUIV="REFRESH" CONTENT=10 or similar HTML The Definitive Guide
DLCTL_NO_DLACTIVEXCTLS
On = Do not download any ActiveX Controls in the document.
DLCTL_NO_FRAMEDOWNLOAD
On = Do not download frames but do download and parse the frameset page. Ignore the frameset, and render no frame tags.
DLCTL_NO_JAVA
On = Do not execute any Java applets.
DLCTL_NO_METACHARSET
On = Suppress HTML Character Sets reflected by meta elements in the document.
DLCTL_NO_RUNACTIVEXCTLS
On = Do not execute any ActiveX Controls in the document.
DLCTL_NO_SCRIPTS
On = Do not execute any scripts.
DLCTL_OFFLINE
Same as DLCTL_OFFLINEIFNOTCONNECTED.
DLCTL_OFFLINEIFNOTCONNECTED
On = Operate in offline mode if not connected to the Internet. This causes the BINDF_GETFROMCACHE_IF_NET_FAIL flag to be set if the computer is connected to the Internet when making requests through URLMON.
DLCTL_PRAGMA_NO_CACHE
On = Do not use Proxy's cache. Instead, force the request through to the server and ignore the proxy, even if the proxy indicates that the data is up to date. This causes the BINDF_PRAGMA_NO_CACHE flag to be set when making requests through URLMON.
DLCTL_RESYNCHRONIZE
On = The browsing component will ignore what is in the cache and ask the server for updated information. The cached information will be used if the server indicates that the cached information is up to date. This causes the BINDF_RESYNCHRONIZEflag to be set when making requests through URLMON. (Here "the cache" presumably refers to local cache, as opposed to a proxy cache involved in DLCTL_PRAGMA_NO_CACHE.)
DLCTL_SILENT
On = Do not display any user interface. This causes the BINDF_SILENTOPERATION flag to be set when making requests through URLMON. (I am hoping this means do not show message boxes and so on.)
DLCTL_URL_ENCODING_DISABLE_UTF8
On = Disable UTF-8 encoding.
DLCTL_URL_ENCODING_ENABLE_UTF8
On = Enable UTF-8 encoding. (What happens if both Enable and Disable are On, or both Off?)
DLCTL_VIDEOS
On = Play any video clips that are contained in the document.

Communication involved in Ambient Properties

Delphi's TWebBrowser implements a container for MS's WebBrowser container. We want to revise TWebBrowser so that it implements the host part of the communication required for WebBrowser to request ambient property values. So our first task is to understand the requirements and steps in that communication.
I found numerous discussions of this topic on the web, at MSDN and elsewhere, but all seemed to be muddled, off-target or incomplete in one way or another, or obfuscated by arcane COM terminology. I derived the following from an example called walkall.cpp (see references), and this turned out to be sufficient to implement the necessary additions to TWebBrowser. Hopefully this will clarify the matters for others too.
As a preliminary, for ambient properties, the container (in our case TWebBrowser or descendant) must implement the following interfaces and methods:
Interface
Method
In TWebBrowser, implemented by
Comment
IOleClientSite

Ancestor TOleControl
Provided by TWebBrowser to WebBrowser as first method of communicating.

QueryInterface
Ancestor TOleControl
Needs to supply TWebBrowser's IDispatch to WebBrowser
IDispatch

Ancestor TOleControl


Invoke
Ancestor TOleControl
We will need to override this method's implementation
In short, TWebBrowser already has implementations of the needed interfaces, but we need to revise TWebBrowser's inherited IDispatch.Invoke so that it can respond to WebBrowser's requests for ambient properties, or at least the ambient property of most interest, DLCONTROL.
Here the communication steps are shown as an activity diagram in tabular format:
Step
Host action
Host pseudo code
___
MSHTML pseudo code
MSHTML action
Create
Instantiate MSHTML
HTMLDoc := get a IHTMLDocument2
-->
<--
  • Create MSHTML
  • Return IHTMLDocument2 interface.

Initialization

MyOLEObj = HTMLDoc as IOLEObject
<--
  • Handle QueryInterface
  • Return IOLEObject



MyOLEObj.SetClientSite(host's IOleClientSite)
-->
Remember HostOleClientSite


Get MSHTML's IOLEControl
MyOLECtl = HTMLDoc as IOLEControl
-->
  • Handle QueryInterface
  • Return IOLEControl



MyOLECtl.OnAmbientPropertyChange
-->
OK, will do!



  • Handle QueryInterface
  • Return IDispatch
<--
-->
HostIDispatch = HostOleClientSite.QI(IDispatch)



  • Handle IDispatch.Invoke
  • Return DLCONTROL value
<--
HostIDispatch.Invoke(DISPID_AMBIENT_ DLCONTROL...)
Get ambient property values

Agenda for TGwWebBrowserTamed

Based on the interactions just described, the complete agenda for our revised control is:
Item
Description
Add AmbientDLCtlSet property to TWebBrowser
  • Add a published property to TWebBrowser which can be set at designtime or runtime. Most conveniently this should be of type set, so that the individual settings can be switched individually in the IDE.
  • Add getter and setter methods.
  • Setter should call WebBrowser's OnAmbientPropertyChange method to prompt WebBrowser to fetch new value.
Revise IDispatch.Invoke
  • Add override for TWebBrowser.Invoke which will add handling for DispID = DISPID_AMBIENT_DLCONTROL, otherwise call inherited Invoke
Diagnostic flag
  • Because I wasn't entirely confident in whether or when WebBrowser would respond to OnAmbientPropertyChange with a call back for values, I added a flag by which to monitor this.

Component and sample code

The requirements just listed are implemented in the code available for download here:
Item
Version
Download link
Comment
GwWebBrowserTamedXE2
2012-02-04






Features included

Project
Description
GwWebBrowserTamedPkg160.bpl
TGwWebBrowserTamed component which can be installed into the IDE. The project is for Delphi XE2 but should require only changes of project settings to work with previous versions of Delphi.
WebBrowserTest01.exe
  • Sample program demonstrating creating TGwWebBrowserTamed at runtime. This is useful to try the component without installing, or to make quick modifications.
  • Includes checkboxes in the UI for exercising each of the DLControl settings live
WebBrowserTest02.exe
  • Sample program demonstrating creating TGwWebBrowserTamed at designtime. This is useful to try out the designtime settings.
  • Includes checkboxes in the UI for exercising each of the DLControl settings live
  • Exe included (see Demos\WebBrowserTest02\Win32\Release)
TGwWebBrowserTamed01.gif
Instructions
  • To browse the web: Enter a URL into the edit slot, and then press Navigate.
  • To change DLCONTROL settings, change some checkboxes and then press "Push to WB"

References

  • MSDN
  • mshtmdid.h Header file listing DISPID_xxx and DLCTL_xxx constants
  • ActiveX Programming Unleashed
    • Chapter 6
      • Includes a primer on Control/Container interaction, and discusses IOleControlSite and IOleClientSite.
      • Discusses ambient properties, including:
        • "The container lays out the contained control, manages keyboard interaction, enables the properties of controls to be saved in some persistent format, handles events generated by controls, and exposes ambient properties to controls. Ambient properties allow OLE controls to retrieve information about the control site provided by their container."
        • "[...] Likewise, a container-provided IDispatch enables controls to access container ambient properties and alert the container of events."
        • "container ambient properties are provided by an IDispatch interface that can be retrieved from the IOleControlSite interface that hosts the control" (Note: "retrieved from" = using QueryInterface, not the IDispatch that can be obtained from IOleControlSite.GetExtendedControl
  • delphidabbler (Peter Johnson) Several relevant articles with sample code
  • TWebBrowserEx, WBComp
    • Source and dcus supplied with XE2, possibly since XE? Eg: $(BDS)\Source\Internet\WebBrowserEx.pas
    • Not sure what these are, as they aren't components installed in the IDE. Are they working code, or vestigial? Maybe part of a tool application? Appear to tackle operating TWebBrowser in edit mode.
    • Demonstrates how to override TOleControl's IDispatch.Invoke, to incorporate the DLCTL_xxx ambient properties, and also provide other responses.

Appendices

Using MSHTML directly instead of WebBrowser

  • Sample code "walkall.exe" (self-extracting archive): http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=944
  • "MSHTML" is discussed in narrative as though it is a tangible class or object: "use of MSHTML", "as MSHTML loads the document", "MSHTML's READYSTATE" etc. While it may be a coherent thing behind the scenes, it's actually engaged by:
    • IHTMLDocument2* m_pMSHTML;
    • CoCreateInstance( CLSID_HTMLDocument, [....] IID_IHTMLDocument2, (LPVOID*)&g_pApp->m_pMSHTML )
    • ... and has a bunch of interfaces, obtainable via the usual COM QueryInterface apparatus.
  • Loading and parsing HTML using MSHTML. 3rd way.
  • MSHTML will do all the behaviors that we didn't want WebBrowser to do, such as loading files, executing scripts etc.
    • The walkall.exe comments explain how to disable all that...
  • But wait! Looks like there's already a TMSHTMLParser hiding in WBComp.pas! I wonder if this is operational?

Using IMarkupServices directly instead of WebBrowser

Jim Beveridge "report from the front" using MSHTML

  • I belatedly discovered this article on Jim Beveridge's blog: How to load MSHTML data He evidently discovered many of the same articles as I have, but has gone further (and probably smarter) implementing several solutions, and run into various difficulties.
  • JB tried a variety of different methods to employ larger and smaller chunks of MSHTML to implement html-to-editable-DOM. My summaries of his notes, as reminders to myself:
  • 1. IMarkupServices.ParseString
    • ParseString wants Unicode. Instead use ParseGlobal for any character set..
    • Returned IHtmlDocument2 object not fully functional. Lacks serialization.
    • IDocument3.getDocumentElement fails.
  • 2. IHtmlDocument2.write()
    • write() requires Unicode.
    • Needs a SAFEARRAY at some point?
  • 3. IPersistStreamInit.Load() to load HTML (get IPersistStreamInit from QI of any interface on MSHTML, I assume)
    • Loading HTML content from a Stream
    • Requires CoInitializeEx(NULL,COINIT_MULTITHREADED)
    • Recent versions of MSHTML require a message loop to get the work done.
    • Bug in MIME interpretation, but fixed in IE7 and later
  • 4. IPersistFile.Load() (interface to HTML doc)
    • Could load html file from memory, if it was in a file. But also requires message loop.
  • 5. IPersistMoniker to feed the stream to MSHTML
    • No need for Unicode buffer, can use in-memory data
My digestion: Aside from trying to troubleshoot and work around actual bugs, JB's main concerns seem to have been:
  • Guaranteeing that the html provided to MSHTML was actually Unicode (where the MSHTML interface required that).
  • Contending with MSHTML (at least some versions of) performing work asynchronously, and thus requiring a message loop.